Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcing Environments
Conventional incentive mechanisms were designed for business environments involving static business processes and a limited number of actors. They are not easily applicable to crowdsourcing and other social computing platforms, characterized by dynamic collaboration patterns and high numbers of actors, because the effects of incentives in these environments are often unforeseen and more costly than in a well-controlled environment of a traditional company.
In this paper we investigate how to design and calibrate incentive schemes for crowdsourcing processes by simulating joint effects of a combination of different participation and incentive mechanisms applied to a working crowd. More specifically, we present a simulation model of incentive schemes and evaluate it on a relevant real-world scenario. We show how the model is used to simulate different compositions of incentive mechanisms and model parameters, and how these choices influence the costs on the system provider side and the number of malicious workers.
Keywordsrewards incentives crowdsourcing social computing collective adaptive systems
Unable to display preview. Download preview PDF.
- 1.Adar, E.: Why I Hate Mechanical Turk Research (and Workshops). Control (2011), http://www.cond.org/eadar-crowdsourcing-workshop.pdf
- 5.Gilbert, N., Troitzsch, K.: Simulation for the social scientist. Open University Press, McGraw-Hill Education (2005)Google Scholar
- 6.Heckman, J., Smith, J.: What do bureaucrats do? The effects of performance standards and bureaucratic preferences on acceptance into the JTPA program. Advances in the Study of Entrepreneurship Innovation and Economic Growth 7, 191–217 (1996)Google Scholar
- 7.Kraus, S., Hoz-Weiss, P., Wilkenfeld, J., Andersen, D.R., Pate, A.: Resolving crises through automated bilateral negotiations. Artificial Intelligence 172(1), 1–18 (2008), http://www.sciencedirect.com/science/article/pii/S0004370207001051 CrossRefzbMATHMathSciNetGoogle Scholar
- 8.Laffont, J.J., Martimort, D.: The Theory of Incentives. Princeton University Press, New Jersey (2002)Google Scholar
- 9.Little, G., Chilton, L.B., Goldman, M., Miller, R.: Exploring iterative and parallel human computation processes. In: Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA 2010, p. 4309 (2010), http://portal.acm.org/citation.cfm?doid=1753846.1754145
- 10.Macal, C.M., North, M.J.: Agent-based modeling and simulation. In: Proceedings of the 2009 Winter Simulation Conference (WSC), pp. 86–98 (December 2009), http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5429318
- 11.Mason, W., Watts, D.J.: Financial incentives and the performance of crowds. In: Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP 2009), vol. 11, pp. 77–85. ACM, Paris (May 2009), http://portal.acm.org/citation.cfm?doid=1809400.1809422 CrossRefGoogle Scholar
- 14.Rockenbach, B., Milinski, M.: The efficient interaction of indirect reciprocity and costly punishment. Nature 444(7120), 718–723, http://dx.doi.org/10.1038/nature05229
- 21.Wegner, R.: Multi-agent malicious behaviour detection. Phd thesis, University of Manitoba (2012), http://hdl.handle.net/1993/9673