Skip to main content
SpringerLink
Log in

Two Sides of Collective Decision Making - Votes from Crowd and Knowledge from Experts

  • Conference paper
  • First Online:
Decision Support Systems X: Cognitive Decision Support Systems and Technologies (ICDSST 2020)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 384))

Included in the following conference series:

Abstract

This paper deals with the role of experts and crowds in solving important societal issues. The authors argue that both experts and crowds are important stakeholders in collective decision making which should jointly participate in the decision-making process to improve it. Usually studied in different research areas, there have been a few models that integrate crowds and experts in a joint model. The authors give an overview of the advantages and disadvantages of crowd and expert decision making and highlight possibilities to connect these two worlds. They position the research in the area of Computational Social Choice (COMSOC) and crowd voting, emerging fields that bring great potential for collective decision making. COMSOC focuses on improving social welfare and the quality of products and services through the inclusion of community or clients into the decision-making process. Despite these altruistic goals, there are several shortcomings that call for the engagement of experts in voting procedures. The authors propose a simple participatory model for weighting and selection of voters and votes through the integration of expert rankings into crowd voting systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Baumeister, D., Rothe, J., Selker, A-K.: Strategic behavior in judgment aggregation. In: Endriss, U. (ed.) Trends in Computational Social Choice, pp. 145–168. AI Access (2017)

    Google Scholar 

  2. Cornelio, C., Pini, M.S., Rossi, F., Venable, K.B.: Multi-agent soft constraint aggregation via sequential voting: theoretical and experimental results. Auton. Agent. Multi-Agent Syst. 33, 159–191 (2019)

    Article  Google Scholar 

  3. Endriss, U., Obraztsova, S., Polukarov, M., Rosenschein, J.S.: Strategic voting with incomplete information. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI 2016), pp. 236–242 (2016)

    Google Scholar 

  4. Faliszewski, P., Reisch, Y., Rothe, J., Schend, L.: Complexity of manipulation, bribery, and campaign management in Bucklin and fallback voting. Auton. Agent. Multi-Agent Syst. 29, 1091–1124 (2015)

    Article  Google Scholar 

  5. Faliszewski, P., Rothe, J.: Control and bribery in voting. In: Brandt, F., Conitzer, V., Endriss, U., Lang, J., Procaccia, A.D. (eds.) Handbook of Computational Social Choice, pp. 145–168. Cambridge University Press, New York (2016)

    Google Scholar 

  6. Skowron, P., Yu, L., Faliszewski, P., Elkind, E.: The complexity of fully proportional representation for single-crossing electorates. Theoret. Comput. Sci. 569, 43–57 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  7. Airiau, S., Endriss, U., Grandi, U., Porello, D., Uckelman, J.: Aggregating dependency graphs into voting agendas in multi-issue elections. In: Walsh, T. (ed.), IJCAI-11: Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence: Barcelona, Catalonia, Spain, 16–22 July 2011, vol. 1, pp. 18–23.: AAAI Press/International Joint Conferences on Artificial Intelligence, Menlo Park (2011)

    Google Scholar 

  8. Lee, D.T., Goel, A., Aitamurto, T., Landemore, H.: Crowdsourcing for participatory democracies: efficient elicitation of social choice functions. In: Proceedings of the Second AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2014), pp. 133–142 (2014)

    Google Scholar 

  9. Slavkovik, M.: Collective decision making with judgment aggregation. In: Computational Decision Making and Data Science Workshop (CDMDSW 2018), Belgrade, Serbia (2018). http://cdmdsw2018.fon.bg.ac.rs/wp-content/uploads/2019/05/marija_slavkovik_collective_decision_making_with_judgment_aggregation.pdf. Accessed 22 Dec 2019

  10. Chen, L., Xu, P., Liu, D.: The effect of crowd voting on participation in crowdsourcing contests. Working paper (2019). 39 pages

    Google Scholar 

  11. Bennet, A., Bennet, D.: The decision-making process for complex situations in a complex environment. In: Burstein, F., Holsapple, C.W. (eds.) Handbook on Decision Support Systems 1, pp. 3–20. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-48713-5_1

    Chapter  Google Scholar 

  12. Yu, C., Chai, Y., Liu, Y.: Literature review on collective intelligence: a crowd science perspective. Int. J. Crowd Sci. 2(1), 64–73 (2018)

    Article  Google Scholar 

  13. Mandic, K., Bobar, V., Delibašić, B.: Modeling interactions among criteria in MCDM methods: a review. In: Delibašić, B., et al. (eds.) ICDSST 2015. LNBIP, vol. 216, pp. 98–109. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18533-0_9

    Chapter  Google Scholar 

  14. Rossi, F.: Preferences, constraints, uncertainty, and multi-agent scenarios. In: ISAIM (2008)

    Google Scholar 

  15. Jackson, S.E.: The consequences of diversity in multidisciplinary work teams. In: West, M.A. (ed.) Handbook of Work Group Psychology, pp. 53–75. Wiley, Chichester (1996)

    Google Scholar 

  16. Miller, N.R.: Reflections on Arrow’s theorem and voting rules. Public Choice 179, 113–124 (2019)

    Article  Google Scholar 

  17. Herings, P.J., Houba, H.: The Condorcet paradox revisited. Soc. Choice Welfare 47, 141–186 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  18. Dodevska, Z.A.: Computational social choice and challenges of voting in multi-agent systems. Tehnika 74(5), 724–730 (2019)

    Article  Google Scholar 

  19. Chevaleyre, Y., Endriss, U., Lang, J., Maudet, N.: A short introduction to computational social choice. In: van Leeuwen, J., Italiano, Giuseppe F., van der Hoek, W., Meinel, C., Sack, H., Plášil, F. (eds.) SOFSEM 2007. LNCS, vol. 4362, pp. 51–69. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-69507-3_4

    Chapter  Google Scholar 

  20. Endriss, U.: Social choice theory as a foundation for multiagent systems. In: Müller, J.P., Weyrich, M., Bazzan, A.L.C. (eds.) MATES 2014. LNCS (LNAI), vol. 8732, pp. 1–6. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11584-9_1

    Chapter  Google Scholar 

  21. Conte, R., Gilbert, N., Bonelli, G., et al.: Manifesto of computational social science. Eur. Phy. J. Spec. Topics 214(1), 325–346 (2012)

    Article  Google Scholar 

  22. Brandt, F., Conitzer, V., Endriss, U., Lang, J., Procaccia, A.D.: Introduction to computational social choice. In: Brandt, F., Conitzer, V., Endriss, U., Lang, J., Procaccia, A.D (eds.). Handbook of Computational Social Choice, pp. 1–20. Cambridge University Press, New York (2016)

    Google Scholar 

  23. Aitamurto, T., Landemore, H., Galli, J.S.: Unmasking the crowd: participants’ motivation factors, expectations, and profile in a crowdsourced law reform. Inf. Commun. Soc. 20(8), 1239–1260 (2017)

    Article  Google Scholar 

  24. Goel, A., Krishnaswamy, A. K., Sakshuwong, S., Aitamurto, T.: Knapsack voting for participatory budgeting. ACM Trans. Econ. Comput. (TEAC) 7(2), (2019)

    Google Scholar 

  25. Ghezzi, A., Gabelloni, D., Martini, A., Natalicchio, A.: Crowdsourcing: a review and suggestions for future research. Int. J. Manag. Rev. 20(2), 343–363 (2017)

    Article  Google Scholar 

  26. Isinkaye, F.O., Folajimi, Y.O., Ojokoh, B.A.: Recommendation systems: principles, methods and evaluation. Egyptian Inf. J. 16(3), 261–273 (2015)

    Article  Google Scholar 

  27. Prpić, J., Shukla, P.P., Kietzmann, J.H., McCarthy, I.P.: How to work a crowd: developing crowd capital through crowdsourcing. Bus. Horiz. 58(1), 77–85 (2015)

    Article  Google Scholar 

  28. Aitamurto, T.: Crowdsourcing for democracy: a new era in policy-making. Publications of the Committee for the Future, Parliament of Finland 1/2012. Helsinki, Finland (2012)

    Google Scholar 

  29. Botan, S., Novaro, A., Endriss, U.: Group manipulation in judgment aggregation. In: Thangarajah, J., Tuyls, K., Jonker, C., Marsella, S. (eds.). Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016), pp. 411–419, Singapore (2016)

    Google Scholar 

  30. Dey, P., Misra, N., Narahari, Y.: Frugal bribery in voting. Theoret. Comput. Sci. 676, 15–32 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  31. Hata, K., Krishna, R., Fei-Fei, L., Bernstein, M.: A glimpse far into the future: understanding long-term crowd worker quality. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 889–901. ACM DL, Portland (2017)

    Google Scholar 

  32. Livshits, B., Mytkowicz, T.: Saving money while polling with InterPoll using power analysis. In: Proceedings of the Second AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2014), pp. 159–170. AAAI Publications (2014)

    Google Scholar 

  33. Gaikwad, S.N.S., et al.: Boomerang: rebounding the consequences of reputation feedback on crowdsourcing platforms. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ACM DL, Tokyo, Japan, pp. 625–637 (2016)

    Google Scholar 

  34. Göritz, A.S., Borchert, K., Hirth, M.: Using attention testing to select crowdsourced workers and research participants. Soc. Sci. Comput. Rev. (2019)

    Google Scholar 

  35. Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., Allahbakhsh, M.: Quality control in crowdsourcing: a survey of quality attributes, assessment techniques, and assurance actions. ACM Comput. Surv. (CSUR) 51(1) (2018). Article 7, 40 p.

    Google Scholar 

  36. Shah, N., Zhou, D.: No oops, you won’t do it again: mechanisms for self-correction in crowdsourcing. In: Proceedings of The 33rd International Conference on Machine Learning (ICML), vol. 48, pp. 1–10. New York (2016)

    Google Scholar 

  37. Kamangar, Z.U., Kamangar, U.A., Ali, Q., Farah, I., Nizamani, S., Ali, T. H.: To enhance effectiveness of crowdsource software testing by applying personality types. In: Proceedings of the 8th International Conference on Software and Information Engineering, pp. 15–19, Cairo, Egypt. ACM DL (2019)

    Google Scholar 

  38. Colman, D.E., Vineyard, J., Letzring, T.D.: Exploring beyond simple demographic variables: differences between traditional laboratory samples and crowdsourced online samples on the Big Five personality traits. Personality Individ. Differ. 133, 41–46 (2018)

    Article  Google Scholar 

  39. Naroditskiy, V., Rahwan, I., Cebrian, M., Jennings, N.R.: Verification in referral-based crowdsourcing. PLoS ONE 7(10), e45924 (2012)

    Article  Google Scholar 

  40. Naroditskiy, V., Stein, S., Tonin, M., Tran-Thanh, L., Vlassopoulos, M., Jennings, N.R.: Referral incentives in crowdfunding. In: Proceedings of the Second AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2014), pp. 171–183. AAAI (2014)

    Google Scholar 

  41. Hung, N.Q.V., Thang, D.C., Weidlich, M., Aberer, K.: Minimizing efforts in validating crowd answers. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia, pp. 999–1014. ACM DL (2015)

    Google Scholar 

  42. Aitamurto, T., Landemore, H.E.: Five design principles for crowdsourced policymaking: assessing the case of crowdsourced off-road traffic law in Finland. J. Soc. Media Organ. 2(1), 1–19 (2015)

    Google Scholar 

  43. Chen, L., Huang, Z., Liu, D.: Pure and hybrid crowds in crowdfunding markets. Financ. Innovation 2, 19 (2016)

    Article  Google Scholar 

  44. Aziz, H., Brandt, F., Elkind, E., Skowron, P.: Computational social choice: the first ten years and beyond. In: Steffen, B., Woeginger, G. (eds.) Computing and Software Science. LNCS, vol. 10000, pp. 48–65. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-91908-9_4

    Chapter  Google Scholar 

  45. Faliszewski, P., Skowron, P., Slinko, A., Talmon, N.: Committee scoring rules: axiomatic classification and hierarchy. In: Kambhampati, S. (ed.), Proceedings of the 25th International Joint Conference on Artificial Intelligence, Palo Alto, California USA, pp. 250–256, AAAI Press (2016)

    Google Scholar 

  46. Endriss, U.: Computational social choice: prospects and challenges. Procedia Comput. Sci. 7, 68–72 (2011)

    Article  Google Scholar 

  47. Gonzalez-Fernandez, Y., Hamidi, S., Chen, S., Liaskos, S.: Efficient elicitation of software configurations using crowd preferences and domain knowledge. Autom. Software Eng. 26, 87–123 (2019)

    Article  Google Scholar 

  48. Moreno-Jiménez, J.M., Polasek, W.: E-democracy and knowledge. A multicriteria framework for the new democratic era. J. Multi-criteria Decis. Anal. 12, 163–176 (2003)

    Google Scholar 

  49. Haan, M.A., Dijkstra, S.G., Dijkstra, P.T.: Expert judgment versus public opinion–evidence from the Eurovision Song Contest. J. Cult. Econ. 29(1), 59–78 (2005)

    Article  Google Scholar 

  50. Mollick, E., Nanda, R.: Wisdom or madness? Comparing crowds with expert evaluation in funding the arts. In: Management Science Articles in Advance, ©2015 INFORMS, pp. 1–21 (2015)

    Google Scholar 

  51. Knecht, K., Stefanescu, D.A., Koenig, R.: Citizen Engagement through design space exploration: integrating citizen knowledge and expert design in computational urban planning. In: Sousa, J.P., Castro Henriques, G., Xavier, J.P. (eds.) Architecture in the Age of the 4th Industrial Revolution: eCAADe SIGraDi 2019, vol. 1, pp. 785–794, eCAADe; SIGraDi; FAUD (2019)

    Google Scholar 

  52. Drapeau, R., Chilton, L. B., Bragg, J., Weld, D.S.: Microtalk: using argumentation to improve crowdsourcing accuracy. In: Fourth AAAI Conference on Human Computation and Crowdsourcing, September 2016

    Google Scholar 

  53. Shams, B., Haratizadeh, S.: Graph-based collaborative ranking. Expert Syst. Appl. 67, 59–70 (2017)

    Article  MATH  Google Scholar 

  54. Shams, B., Haratizadeh, S.: Reliable graph-based collaborative ranking. Inf. Sci. 432, 116–132 (2018)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This paper is a result of the project ONR - N62909-19-1-2008 supported by the Office for Naval Research, the United States: Aggregating computational algorithms and human decision-making preferences in multi-agent settings.

Author information

Authors and Affiliations

  1. Faculty of Organizational Sciences, University of Belgrade, 154 Jove Ilića, 11000, Belgrade, Serbia

    Zorica A. Dodevska, Ana Kovacevic, Milan Vukicevic & Boris Delibašić

Authors
  1. Zorica A. Dodevska
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ana Kovacevic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Milan Vukicevic
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Boris Delibašić
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zorica A. Dodevska .

Editor information

Editors and Affiliations

  1. University of Zaragoza, Zaragoza, Spain

    José María Moreno-Jiménez

  2. University of Namur, Namur, Belgium

    Isabelle Linden

  3. SimTech Simulation Technology, Graz, Austria

    Fatima Dargam

  4. Loughborough University, Loughborough, UK

    Uchitha Jayawickrama

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dodevska, Z.A., Kovacevic, A., Vukicevic, M., Delibašić, B. (2020). Two Sides of Collective Decision Making - Votes from Crowd and Knowledge from Experts. In: Moreno-Jiménez, J., Linden, I., Dargam, F., Jayawickrama, U. (eds) Decision Support Systems X: Cognitive Decision Support Systems and Technologies. ICDSST 2020. Lecture Notes in Business Information Processing, vol 384. Springer, Cham. https://doi.org/10.1007/978-3-030-46224-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-46224-6_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-46223-9

  • Online ISBN: 978-3-030-46224-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation