Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcing Environments

  • Ognjen Scekic
  • Christoph Dorn
  • Schahram Dustdar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8185)


Conventional incentive mechanisms were designed for business environments involving static business processes and a limited number of actors. They are not easily applicable to crowdsourcing and other social computing platforms, characterized by dynamic collaboration patterns and high numbers of actors, because the effects of incentives in these environments are often unforeseen and more costly than in a well-controlled environment of a traditional company.

In this paper we investigate how to design and calibrate incentive schemes for crowdsourcing processes by simulating joint effects of a combination of different participation and incentive mechanisms applied to a working crowd. More specifically, we present a simulation model of incentive schemes and evaluate it on a relevant real-world scenario. We show how the model is used to simulate different compositions of incentive mechanisms and model parameters, and how these choices influence the costs on the system provider side and the number of malicious workers.


rewards incentives crowdsourcing social computing collective adaptive systems 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adar, E.: Why I Hate Mechanical Turk Research (and Workshops). Control (2011),
  2. 2.
    Bloom, M., Milkovich, G.: The relationship between risk, incentive pay, and organizational performance. The Academy of Management Journal 41(3), 283–297 (1998), CrossRefGoogle Scholar
  3. 3.
    Dorn, C., Edwards, G., Medvidovic, N.: Analyzing design tradeoffs in large-scale socio-technical systems through simulation of dynamic collaboration patterns. In: Meersman, R., et al. (eds.) OTM 2012, Part I. LNCS, vol. 7565, pp. 362–379. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  4. 4.
    Gal, Y., Grosz, B., Kraus, S., Pfeffer, A., Shieber, S.: Agent decision-making in open mixed networks. Artif. Intell. 174(18), 1460–1480 (2010), CrossRefMathSciNetGoogle Scholar
  5. 5.
    Gilbert, N., Troitzsch, K.: Simulation for the social scientist. Open University Press, McGraw-Hill Education (2005)Google Scholar
  6. 6.
    Heckman, J., Smith, J.: What do bureaucrats do? The effects of performance standards and bureaucratic preferences on acceptance into the JTPA program. Advances in the Study of Entrepreneurship Innovation and Economic Growth 7, 191–217 (1996)Google Scholar
  7. 7.
    Kraus, S., Hoz-Weiss, P., Wilkenfeld, J., Andersen, D.R., Pate, A.: Resolving crises through automated bilateral negotiations. Artificial Intelligence 172(1), 1–18 (2008), CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Laffont, J.J., Martimort, D.: The Theory of Incentives. Princeton University Press, New Jersey (2002)Google Scholar
  9. 9.
    Little, G., Chilton, L.B., Goldman, M., Miller, R.: Exploring iterative and parallel human computation processes. In: Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA 2010, p. 4309 (2010),
  10. 10.
    Macal, C.M., North, M.J.: Agent-based modeling and simulation. In: Proceedings of the 2009 Winter Simulation Conference (WSC), pp. 86–98 (December 2009),
  11. 11.
    Mason, W., Watts, D.J.: Financial incentives and the performance of crowds. In: Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP 2009), vol. 11, pp. 77–85. ACM, Paris (May 2009), CrossRefGoogle Scholar
  12. 12.
    Milinski, M., Semmann, D., Krambeck, H.J.: Reputation helps solve the ‘tragedy of the commons’. Nature 415(6870), 424–426 (2002), CrossRefGoogle Scholar
  13. 13.
    Prendergast, C.: The provision of incentives in firms. Journal of Economic Literature 37(1), 7–63 (1999), CrossRefGoogle Scholar
  14. 14.
    Rockenbach, B., Milinski, M.: The efficient interaction of indirect reciprocity and costly punishment. Nature 444(7120), 718–723,
  15. 15.
    Scekic, O., Truong, H.-L., Dustdar, S.: Modeling rewards and incentive mechanisms for social BPM. In: Barros, A., Gal, A., Kindler, E. (eds.) BPM 2012. LNCS, vol. 7481, pp. 150–155. Springer, Heidelberg (2012), CrossRefGoogle Scholar
  16. 16.
    Scekic, O., Truong, H.-L., Dustdar, S.: Incentives and rewarding in social computing. Commun. ACM 56(6), 72–82 (2013), CrossRefGoogle Scholar
  17. 17.
    Scekic, O., Truong, H.-L., Dustdar, S.: Programming incentives in information systems. In: Salinesi, C., Norrie, M.C., Pastor, O. (eds.) CAiSE 2013. LNCS, vol. 7908, pp. 688–703. Springer, Heidelberg (2013), CrossRefGoogle Scholar
  18. 18.
    Semmann, D., Krambeck, H.J., Milinski, M.: Strategic investment in reputation. Behavioral Ecology and Sociobiology 56(3), 248–252 (2004), CrossRefGoogle Scholar
  19. 19.
    Tokarchuk, O., Cuel, R., Zamarian, M.: Analyzing crowd labor and designing incentives for humans in the loop. IEEE Internet Computing 16, 45–51 (2012)CrossRefGoogle Scholar
  20. 20.
    Wedekind, C., Milinski, M.: Cooperation Through Image Scoring in Humans. Science 288(5467), 850–852 (2000), CrossRefGoogle Scholar
  21. 21.
    Wegner, R.: Multi-agent malicious behaviour detection. Phd thesis, University of Manitoba (2012),

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Ognjen Scekic
    • 1
  • Christoph Dorn
    • 1
  • Schahram Dustdar
    • 1
  1. 1.Distributed Systems GroupVienna University of TechnologyAustria

Personalised recommendations