Skip to main content
SpringerLink
Log in

Crowdsourcing and Human Computation, Introduction

  • Reference work entry
  • First Online:
Encyclopedia of Social Network Analysis and Mining

Synonyms

Crowdsourcing; Human computation

Glossary

AC:

Automatic computers

AI:

Artificial intelligence

AMT:

Amazon Mechanical Turk

GWAP:

Games with a purpose

HIT:

Human intelligence task

IR:

Information retrieval

MT:

Machine translation

NLP:

Natural language processing

Introduction

The first computers were actually people (Grier 2005). Later, machines were built, known at the time as Automatic computers (ACs), to perform many routine computations. While such machines have continued to advance and now perform many of the routine processing tasks once delegated to people, human capabilities still continue to exceed state-of-the-art artificial intelligence (AI) on a variety of important data analysis tasks, such as those involving image (Sorokin and Forsyth 2008) and language understanding (Snow et al. 2008). Consequently, today's Internet-based access to 24/7 online human crowds has sparked the advent of crowdsourcing (Howe 2006) and a renaissance of human computation (Quinn and...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 1,500.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 549.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Alonso O (2012) Implementing crowdsourcing-based relevance experimentation: an industrial perspective. Info Retr J Spec Issue Crowdsourc

    Google Scholar 

  • Alonso O, Rose DE, Stewart, B (2008) Crowdsourcing for relevance evaluation. ACM SIGIR Forum 42(2): 9–15

    Google Scholar 

  • Artstein R, Poesio, M (2008) Inter-coder agreement for computational linguistics. Comput Linguist 34(4):555–596

    Google Scholar 

  • Bederson BB, Quinn AJ (2011) Web workers unite! addressing challenges of online laborers. In: CHI workshop on crowdsourcing and human computation. ACM

    Google Scholar 

  • Callison-Burch C (2009) Fast, cheap, and creative: evaluating translation quality using amazon's mechanical turk. In: Proceedings of the 2009 conference on empirical methods in natural language processing: volume 1-Volume 1. Association for Computational Linguistics, pp 286–295

    Google Scholar 

  • Davis J, Arderiu J, Lin H, Nevins Z, Schuon S, Gallo O, Yang M (2010) The HPU. In: Computer vision and pattern recognition workshops (CVPRW), pp 9–16

    Google Scholar 

  • Dawid AP, Skene AM (1979) Maximum likelihood estimation of observer error-rates using the em algorithm. Appl Stat 28(1):20–28

    Google Scholar 

  • Felstiner A (2010) Sweatshop or paper route? Child labor laws and in-game work. In: Proceedings of the 1st annual conference on the future of distributed work (CrowdConf), San Francisco

    Google Scholar 

  • Fort K, Adda G, Cohen KB (2011) Amazon mechanical turk: gold mine or coal mine? Comput Linguist 37(2):413–420

    Google Scholar 

  • Grier DA (2005) When computers were human, vol 316. Princeton University Press, Princeton

    Google Scholar 

  • Horowitz D, Kamvar SD (2010) The anatomy of a large-scale social search engine. In: Proceedings of the 19th international conference on world wide web. ACM, pp 431–440

    Google Scholar 

  • Howe J (2006) The rise of crowdsourcing. Wired Mag 14(6):1–4

    MathSciNet  Google Scholar 

  • Ipeirotis P (2010) Demographics of mechanical turk. Tech. Rep., CeDER-10-01, New York University

    Google Scholar 

  • Irani L, Silberman M (2013) Turkopticon: interrupting worker invisibility in amazon mechanical turk. In: Proceeding of the ACM SIGCHI conference on human factors in computing systems

    Google Scholar 

  • Kazai G, Kamps J, Milic-Frayling N (2012) An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Info Retr J Spec Issue Crowdsourc

    Google Scholar 

  • Kittur A, Nickerson JV, Bernstein M, Gerber E, Shaw A, Zimmerman J, Lease M, Horton J (2013) The future of crowd work. In: Proceedings of the ACM conference on computer supported cooperative work (CSCW), pp 1301–1318

    Google Scholar 

  • Klinger J, Lease M (2011) Enabling trust in crowd labor relations through identity sharing. In: Proceedings of the 74th annual meeting of the American Society for Information Science and Technology (ASIS&T), pp 1–4

    Google Scholar 

  • Kochhar S, Mazzocchi S, Paritosh P (2010) The anatomy of a large-scale human computation engine. In: Proceedings of the ACM SIGKDD workshop on human computation. ACM, pp 10–17

    Google Scholar 

  • Kulkarni A, Gutheim P, Narula P, Rolnitzky D, Parikh T, Hartmann B (2012) Mobileworks: designing for quality in a managed crowdsourcing architecture. IEEE Internet Comput 16(5):28

    Google Scholar 

  • Law E, von Ahn L (2011) Human computation. Synth Lect Artif Intell Mach Learn 5(3):1–121

    Google Scholar 

  • Le J, Edmonds A, Hester V, Biewald L (2010) Ensuring quality in crowdsourced search relevance evaluation: the effects of training question distribution. In: SIGIR 2010 workshop on crowdsourcing for search evaluation, pp 21–26

    Google Scholar 

  • Lease M, Hullman J, Bigham JP, Bernstein MS, Kim J, Lasecki WS, Bakhshi S, Mitra T, Miller RC (2013) Mechanical turk is not anonymous. In: Social science research network (SSRN). Online: http://ssrn.com/abstract=2228728. SSRN ID: 2228728

  • Liu D, Bias R, Lease M, Kuipers R (2012) Crowdsourcing for usability testing. In: Proceedings of the 75th annual meeting of the American Society for Information Science and Technology (ASIS&T)

    Google Scholar 

  • Mason W, Watts DJ (2009) Financial incentives and the performance of crowds. In: SIGKDD

    Google Scholar 

  • Munro R (2012) Crowdsourcing and the crisis-affected community lessons learned and looking forward from mission 4636. Info Retr J Spec Issue Crowdsourc

    Google Scholar 

  • Paritosh P, Ipeirotis P, Cooper M, Suri S (2011) The computer is the new sewing machine: benefits and perils of crowdsourcing. In: Proceedings of the 20th international conference companion on World wide web. ACM, pp 325–326

    Google Scholar 

  • Pickard G, Pan W, Rahwan I, Cebrian M, Crane R, Madan A, Pentland A (2011) Time-critical social mobilization. Science 334(6055):509–512

    Google Scholar 

  • Quinn AJ, Bederson BB (2011) Human computation: a survey and taxonomy of a growing field. In: 2011 annual ACM SIGCHI conference on human factors in computing systems, pp 1403–1412

    Google Scholar 

  • Ross J, Irani L, Silberman M, Zaldivar A, Tomlinson B (2010) Who are the crowdworkers? Shifting demographics in mechanical turk. In: Proceedings of the 28th of the international conference extended abstracts on human factors in computing systems. ACM, pp 2863–2872

    Google Scholar 

  • Sheng V, Provost F, Ipeirotis P (2008) Get another label? improving data quality and data mining using multiple, noisy labelers. In: Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 614–622

    Google Scholar 

  • Silberman M, Irani L, Ross J (2010) Ethics and tactics of professional crowdwork. XRDS: Crossroads ACM Mag Stud 17(2):39–43

    Google Scholar 

  • Snow R, O'Connor B, Jurafsky D, Ng AY (2008) Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In: Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pp 254–263

    Google Scholar 

  • Sorokin A, Forsyth D (2008) Utility data annotation with amazon mechanical turk. In: IEEE computer society conference on computer vision and pattern recognition workshops, 2008 (CVPRW'08). IEEE, pp 1–8

    Google Scholar 

  • Surowiecki J (2005) The wisdom of crowds. Anchor, New York

    Google Scholar 

  • Tang W, Lease M (2011) Semi-supervised consensus labeling for crowdsourcing. In: Proceedings of the ACM SIGIR workshop on crowdsourcing for information retrieval

    Google Scholar 

  • Viégas F, Wattenberg M, Mckeon M (2007) The hidden order of wikipedia. In: Online communities and social computing. Springer, Berlin/New York, pp 445–454

    Google Scholar 

  • Wang J, Ipeirotis P, Provost F (2011) Managing crowdsourcing workers. In: The 2011 winter conference on business intelligence

    Google Scholar 

  • Wolfson S, Lease, M (2011) Look before you leap: legal pitfalls of crowdsourcing. In: Proceedings of the 74th annual meeting of the American Society for Information Science and Technology (ASIS&T)

    Google Scholar 

  • Yan T, Kumar V, Ganesan D (2010) CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones. In: Proceedings of the 8th international conference on mobile systems, applications, and services (MOBISYS). ACM, pp 77–90

    Google Scholar 

  • Zuccon G, Leelanupab T, Whiting S, Yilmaz E, Jose JM, Azzopardi L (2012) Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems. Info Retr J Spec Issue Crowdsourc

    Google Scholar 

Recommended Reading

  • Barr J, Cabrera LF (2006) AI gets a brain. Queue 4(4): 24–29

    Google Scholar 

  • Bederson BB, Quinn A (2011) Participation in human computation. In: CHI workshop on crowdsourcing and human computation. ACM

    Google Scholar 

  • Bell RM, Koren Y (2007) Lessons from the netflix prize challenge. ACM SIGKDD Explor Newsl 9(2):75–79

    Google Scholar 

  • Benkler Y (2002) Coase's penguin, or, linux and “the nature of the firm”. Yale Law J 112(3):369–446

    Google Scholar 

  • Bryant SL, Forte A, Bruckman A (2005) Becoming wikipedian: transformation of participation in a collaborative online encyclopedia. In: Proceedings of the 2005 international ACM SIGGROUP conference on supporting group work. ACM, pp 1–10

    Google Scholar 

  • Chen JJ, Menezes NJ, Bradley AD, North TA (2011) Opportunities for crowdsourcing research on amazon mechanical turk. In: CHI workshop on crowdsourcing and human computation

    Google Scholar 

  • Chi EH, Bernstein MS (2012) Leveraging online populations for crowdsourcing: guest editors' introduction to the special issue. IEEE Internet Comput 16(5):10–12

    Google Scholar 

  • Cushing E (2013) Amazon mechanical turk: the digital sweatshop. UTNE reader. www.utne.com/science-technology/amazon-mechanical-turk-zm0z13jfzlin.as%20px

  • danah boyd (2011) What is the role of technology in human trafficking? http://www.zephoria.org/thoughts/archives/2011/12/07/tech-trafficking.html

  • Dekel O, Shamir O (2008) Learning to classify with missing and corrupted features. In: Proceedings of the 25th international conference on machine learning. ACM, pp 216–223

    Google Scholar 

  • Grady C, Lease M (2010) Crowdsourcing document relevance assessment with mechanical turk. In: Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical turk, Los Angeles. Association for Computational Linguistics, pp 172–179

    Google Scholar 

  • Hecht B, Teevan J, Morris MR, Liebling D (2012) Searchbuddies: bringing search engines into the conversation. In: Proceedings of ICWSM 2012

    Google Scholar 

  • Irwin A (2001) Constructing the scientific citizen: Science and democracy in the biosciences. Public Underst Sci 10(1):1–18

    Google Scholar 

  • Lease M, Yilmaz E (2013) Crowdsourcing for information retrieval: introduction to the special issue. Info Retr 16(4):91–100

    Google Scholar 

  • Levine BN, Shields C, Margolin NB (2006) A survey of solutions to the sybil attack. Tech. Rep., University of Massachusetts Amherst, Amherst

    Google Scholar 

  • McCreadie R, Macdonald C, Ounis I (2012) Crowdterrier: automatic crowdsourced relevance assessments with terrier. In: Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval. ACM, pp 1005–1005

    Google Scholar 

  • Mitchell S (2010) Inside the online sweatshops. In: PC pro magazine. www.pcpro.co.uk/features/360127/inside-the-online-sweatshops

  • Narula P, Gutheim P, Rolnitzky D, Kulkarni A, Hartmann B (2011) Mobileworks: a mobile crowdsourcing platform for workers at the bottom of the pyramid. In: AAAI human computation workshop

    Google Scholar 

  • Oleson D, Sorokin A, Laughlin G, Hester V, Le J, Biewald L (2011) Programmatic gold: targeted and scalable quality assurance in crowdsourcing. In: AAAI workshop on human computation

    Google Scholar 

  • Pontin J, (2007) Artificial intelligence, with help from the humans. New York Times, 25 March 2007

    Google Scholar 

  • Shaw A (2013) Some initial thoughts on the otey vs crowdflower case. http://fringethoughts.wordpress.com/2013/01/09/some-initial-thoughts-on-the-otey-vs-crowdflower-case/

  • Smyth P, Fayyad U, Burl M, Perona P, Baldi P (1995) Inferring ground truth from subjective labelling of venus images. Adv Neural Info Process Syst, 1085–1092

    Google Scholar 

  • Stvilia B, Twidale MB, Smith LC, Gasser L (2008) Information quality work organization in wikipedia. J Am Soc Info Sci Technol 59(6):983–1001

    Google Scholar 

  • Sunstein CR (2006) Infotopia: how many minds produce knowledge. Oxford University Press, Oxford

    Google Scholar 

  • Vincent D (2011) China used prisoners in lucrative internet gaming work. The Guardian, 25 May 2011

    Google Scholar 

  • von Ahn L (2005) Human computation. PhD thesis, Carnegie Mellon University. Tech. Rep., CMU-CS-05-193

    Google Scholar 

  • von Ahn L, Dabbish L (2008) Designing games with a purpose. Commun ACM 51(8):58–67

    Google Scholar 

  • von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) recaptcha: human-based character recognition via web security measures. Science 321(5895):1465–1468

    MATH  MathSciNet  Google Scholar 

  • Wallach H, Vaughan JW (2010) Workshop on computational social science and the wisdom of crowds. In: NIPS

    Google Scholar 

  • Wauthier FL, Jordan MI (2011) Bayesian bias mitigation for crowdsourcing. In: Proceedings of NIPS

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information, University of Texas at Austin, Austin, TX, USA

    Matthew Lease

  2. Microsoft Corp., Microsoft Corp., Mountain View, CA, USA

    Omar Alonso

Authors
  1. Matthew Lease
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Omar Alonso
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Calgary, Calgary, AB, Canada

    Reda Alhajj

  2. Department of Computer Science, University of Calgary, Calgary, AB, Canada

    Jon Rokne

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media New York

About this entry

Cite this entry

Lease, M., Alonso, O. (2014). Crowdsourcing and Human Computation, Introduction. In: Alhajj, R., Rokne, J. (eds) Encyclopedia of Social Network Analysis and Mining. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-6170-8_107

Download citation

Publish with us

Policies and ethics

Search

Navigation