Skip to main content
SpringerLink
Log in

Crowdsourcing Task Marketplaces

  • Chapter
  • First Online:
Book cover Service-Oriented Crowdsourcing

Part of the book series: SpringerBriefs in Computer Science ((BRIEFSCOMPUTER))

Abstract

In this chapter, we discuss detailed statistics of the popular Amazon Mechanical Turk (AMT) marketplace to provide insights in task properties and requester behavior. We present a model to automatically infer requester communities based on task keywords. Hierarchical clustering is used to identify relations between keywords associated with tasks. We present novel techniques to rank communities and requesters by using a graph-based algorithm. Furthermore, we introduce models and methods for the discovery of relevant crowdsourcing brokers who are able to act as intermediaries between requesters and platforms such as AMT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alonso, O., Rose, D.E., Stewart, B.: Crowdsourcing for relevance evaluation. SIGIR Forum 42(2), 9–15 (2008)

    Article  Google Scholar 

  2. Barabasi, A.-L., Albert, R.: Emergence of scaling in random networks. Science 286, 509 (1999)

    Article  MathSciNet  Google Scholar 

  3. Bhattacharyya, P., Garg, A., Wu, S.: Analysis of user keyword similarity in online social networks. Soc. Netw. Anal. Min. 1, 143–158 (2011). doi:10.1007/s13278-010-0006-4

  4. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)

    MATH  Google Scholar 

  5. Branting, L.: Context-sensitive detection of local community structure. Soc. Netw. Anal. Min. 1, 1–11 (2012). doi:10.1007/s13278-011-0035-7

  6. Burt, R.S.: Structural Holes: The Social Structure of Competition. Harvard University Press, Cambridge (1992)

    Google Scholar 

  7. Callison-Burch, C., Dredze M.: Creating speech and language data with amazon’s mechanical turk. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10. Association for Computational Linguistics, pp. 1–12. Stroudsburg, PA, USA (2010)

    Google Scholar 

  8. Carvalho, V.R., Lease, M., Yilmaz, E.: Crowdsourcing for search evaluation. SIGIR Forum 44(2), 17–22 (2011)

    Article  Google Scholar 

  9. Cazabet, R., Takeda, H., Hamasaki, M., Amblard, F.: Using dynamic community detection to identify trends in user-generated content. Soc. Netw. Anal. Min. 1–11 (2012). doi:10.1007/s13278-012-0074-8

  10. Chakrabarti, S.: Dynamic personalized pagerank in entity-relation graphs. In: Proceedings of the 16th International Conference on World Wide Web, WWW ’07, pp. 571–580. ACM, New York (2007)

    Google Scholar 

  11. Chang, J., Boyd-Graber, J., Gerrish, S., Wang, C., Blei, D.: Reading tea leaves: How humans interpret topic models. In: Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C.K.I., Culotta, A. (eds.) Advances in Neural Information Processing Systems 22, pp. 288–296. MIT press, Cambridge (2009)

    Google Scholar 

  12. ClickWorker: http://www.clickworker.com/ (2012). Accessed 20 Aug

  13. CrowdFlower: http://crowdflower.com/ (2012). Accessed 20 Aug

  14. Doan, A., Ramakrishnan, R., Halevy, A.Y.: Crowdsourcing systems on the world-wide web. Commun. ACM 54(4), 86–96 (2011)

    Article  Google Scholar 

  15. Eda, T., Yoshikawa, M., Yamamuro, M.: Locally expandable allocation of folksonomy tags in a directed acyclic graph. In: Proceedings of the 9th International Conference on Web Information Systems Engineering, WISE ’08, pp. 151–162. Springer, Berlin, Heidelberg (2008)

    Google Scholar 

  16. Fazeen, M., Dantu, R., Guturu, P.: Identification of leaders, lurkers, associates and spammers in a social network: context-dependent and context-independent approaches. Soc. Netw. Anal. Min. 1, 241–254 (2011). doi:10.1007/s13278-011-0017-9

  17. Fisher, D., Smith, M., Welser, H.T.: You are who you talk to: detecting roles in usenet newsgroups. In: Proceedings of the 39th Annual Hawaii International Conference on System Sciences, HICSS ’06, Vol. 3, p. 59.2, IEEE Computer Society, Washington, DC, USA (2006)

    Google Scholar 

  18. Flickr: http://www.flickr.com/ (2012). Accessed 20 Aug

  19. Fogaras, D., Rácz, B., Csalogány, K., Sarlós, T.: Towards scaling fully personalized pagerank: algorithms, lower bounds, and experiments. Internet Math. 2(3), 333–358 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  20. Franklin, M.J., Kossmann, D., Kraska, T., Ramesh, S., Xin, R.: Crowddb: answering queries with crowdsourcing. In: Proceedings of the 2011 International Conference on Management of Data, SIGMOD ’11, pp. 61–72. ACM, New York (2011)

    Google Scholar 

  21. Gemmell, J., Shepitsen, A., Mobasher, B., Burke, R.: Personalizing navigation in folksonomies using hierarchical tag clustering. In: Proceedings of the 10th International Conference on Data Warehousing and Knowledge Discovery, DaWaK ’08, pp. 196–205. Springer, Berlin, Heidelberg (2008)

    Google Scholar 

  22. Golder, S., Huberman, B.A.: Usage patterns of collaborative tagging systems. J. Inform. Sci. 32(2), 198–208 (2006)

    Article  Google Scholar 

  23. Haveliwala, T.H.: Topic-sensitive pagerank. In: Proceedings of the 11th International Conference on World Wide Web, WWW ’02, pp. 517–526. ACM, New York (2002)

    Google Scholar 

  24. Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the 28th International Conference on Human factors in Computing Systems, CHI ’10, pp. 203–212. ACM, New York (2010)

    Google Scholar 

  25. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)

    Article  Google Scholar 

  26. Heymann, P., Garcia-Molina, H.: Collaborative creation of communal hierarchical taxonomies in social tagging systems. Technical report, Computer Science Department, Standford University, April (2006)

    Google Scholar 

  27. Howe, J.: The rise of crowdsourcing. Wired 14(14), 1–5 (2006)

    Google Scholar 

  28. Howe, J.: Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business. Crown Business, New York (2008)

    Google Scholar 

  29. Ipeirotis, P.G.: Analyzing the amazon mechanical turk marketplace. XRDS 17, 16–21 (2010)

    Article  Google Scholar 

  30. Jeh, G., Widom, J.: Scaling personalized web search. In: Proceedings of the 12th International Conference on World Wide Web, WWW ’03, pp. 271–279. ACM, New York (2003)

    Google Scholar 

  31. Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical turk. In: Proceedings of the 26th Annual SIGCHI Conference on Human factors in Computing Systems, CHI ’08, pp. 453–456. ACM, New York (2008)

    Google Scholar 

  32. Kleinberg, J.M.: Authoritative sources in a hyperlinked environment. J. ACM 46(5), 604–632 (1999)

    Google Scholar 

  33. Kourtellis, N., Alahakoon, T., Simha, R., Iamnitchi, A., Tripathi, R.: Identifying high betweenness centrality nodes in large social networks. Soc. Netw. Anal. Min. 1–16 (2012). doi:10.1007/s13278-012-0076-6

  34. Lampe, C., Resnick, P.: Slash(dot) and burn: distributed moderation in a large online conversation space. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, pp. 543–550. ACM, New York (2004)

    Google Scholar 

  35. Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23rd Annual ACM symposium on User Interface Software and Technology, UIST ’10, pp. 57–66. ACM, New York (2010)

    Google Scholar 

  36. Marge, M., Banerjee, S., Rudnicky, A.I.: Using the amazon mechanical turk for transcription of spoken language. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 5270–5273, 2010

    Google Scholar 

  37. Michlmayr, E., Cayzer, S.: Learning user profiles from tagging data and leveraging them for personal(ized) information access. In: Tagging and Metadata for Social Information Organization, Workshop, WWW07, 2007

    Google Scholar 

  38. Munro, R., Bethard, S., Kuperman, V., Lai, V.T., Melnick, R., Potts,C., Schnoebelen, T., Tily, H.: Crowdsourcing and language studies: the new generation of linguistic data. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, pp. 122–130. Association for Computational Linguistics, Stroudsburg, PA, USA (2010)

    Google Scholar 

  39. oDesk: http://www.odesk.com/ (2012). Accessed 20 Aug

  40. Page, L., Brin, S., Motwani, R., Winograd, T.: Bringing order to the web. The pagerank citation ranking (1999)

    Google Scholar 

  41. Parameswaran, A., Park, H., Garcia-Molina, H., Polyzotis, N., Widom, J.: Deco: declarative crowdsourcing. Technical report, Stanford University (2011)

    Google Scholar 

  42. Psaier, H., Skopik, F., Schall, D., Dustdar, S.: Resource and agreement management in dynamic crowdcomputing environments. In: EDOC, pp. 193–202. IEEE Computer Society, Washington, DC (2011)

    Google Scholar 

  43. Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, CHI ’11, pp. 1403–1412. ACM, New York (2011)

    Google Scholar 

  44. Romesburg, C.: Cluster Analysis for Researchers. Krieger, Florida (2004)

    Google Scholar 

  45. Rosvall, M., Bergstrom, C.T.: Maps of random walks on complex networks reveal community structure. PNAS 105, 1118 (2008)

    Article  Google Scholar 

  46. Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Inf. Process. Manage. 24(5), 513–523 (1988)

    Article  Google Scholar 

  47. Samasource: http://samasource.org/ (2012). Accessed 20 Aug

  48. Satzger, B., Psaier, H., Schall, D., Dustdar, S.: Stimulating skill evolution in market-based crowdsourcing. In: BPM, pp. 66–82. Springer, Berlin (2011)

    Google Scholar 

  49. Schall, D.: A human-centric runtime framework for mixed service-oriented systems. Distributed and Parallel Databases, 29, 333–360 (2011). doi: 10.1007/s10619-011-7081-z(Springer, Berlin)

  50. Schall, D.: Expertise ranking using activity and contextual link measures. Data Knowl. Eng. 71(1), 92–113 (2012). doi:10.1016/j.datak.2011.08.001

    Article  Google Scholar 

  51. Schall, D., Skopik, F.: An analysis of the structure and dynamics of large-scale q/a communities. In: Eder, J., Bieliková, M., Tjoa, A.M. (eds.) ADBIS, Lecture Notes in Computer Science, vol. 6909, pp. 285–301. Springer, Berlin (2011)

    Google Scholar 

  52. Schall, D., Skopik, F., Psaier, H., Dustdar, S.: Bridging socially-enhanced virtual communities. In: Chu, W.C., Wong, W.E., Palakal, M.J., Hung, C.-C. (eds.) SAC, pp. 792–799. ACM, New York (2011)

    Google Scholar 

  53. Shepitsen, A., Gemmell, J., Mobasher, B., Burke, R.: Personalized recommendation in social tagging systems using hierarchical clustering. In: Proceedings of the 2008 ACM Conference on Recommender Systems, RecSys’08, pp. 259–266. ACM, New York (2008)

    Google Scholar 

  54. Sigurbjörnsson, B., van Zwol, R.: Flickr tag recommendation based on collective knowledge. In: Proceedings of the 17th International Conference on World Wide Web, WWW’08, pp. 327–336. ACM, New York (2008)

    Google Scholar 

  55. Skopik, F., Schall, D., Dustdar, S.: Start trusting strangers? bootstrapping and prediction of trust. In: Vossen, G., Long, D.D.E., Yu, J.X. (eds.) WISE, Lecture Notes in Computer Science, vol. 5802, pp. 275–289. Springer, Berlin (2009)

    Google Scholar 

  56. SmartSheet. http://www.smartsheet.com/ (2012). Accessed 20 Aug

  57. SpeechInk. http://www.speechink.com/ (2012). Accessed 20 Aug

  58. Vukovic, M.: Crowdsourcing for enterprises. In: Proceedings of the 2009 Congress on Services-I, SERVICES ’09, pp. 686–692. IEEE Computer Society, Washington, DC (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Siemens Corporate Technology, Siemensstrasse 90, 1211, Vienna, Austria

    Daniel Schall

Authors
  1. Daniel Schall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Schall .

Rights and permissions

Reprints and permissions

Copyright information

© 2012 The Author(s)

About this chapter

Cite this chapter

Schall, D. (2012). Crowdsourcing Task Marketplaces. In: Service-Oriented Crowdsourcing. SpringerBriefs in Computer Science. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-5956-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-5956-9_2

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-5955-2

  • Online ISBN: 978-1-4614-5956-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation