Information Retrieval

, Volume 16, Issue 2, pp 91–100

Crowdsourcing for information retrieval: introduction to the special issue

Crowd Sourcing

Abstract

This introduction to the special issue summarizes and contextualizes six novel research contributions at the intersection of information retrieval (IR) and crowdsourcing (also overlapping crowdsourcing’s closely-related sibling, human computation). Several of the papers included in this special issue represent deeper investigations into research topics for which earlier stages of the authors’ research were disseminated at crowdsourcing workshops at SIGIR and WSDM conferences, as well as at the NIST TREC conference. Since the first proposed use of crowdsourcing for IR in 2008, interest in this area has quickly accelerated and led to three workshops, an ongoing NIST TREC track, and a great variety of published papers, talks, and tutorials. We briefly summarize the area in order to help situate the contributions appearing in this special issue. We also discuss some broader current trends and issues in crowdsourcing which bear upon its use in IR and other fields.

Keywords

Crowdsourcing Human computation Search evaluation 

References

  1. Alonso, O. (2010). Tutorial: Crowdsourcing for relevance evaluation. In Proceedings of the 32nd European conference on IR research (ECIR). Slides available online at http://ir.ischool.utexas.edu/cse2010/materials/alonso-ecir2010-tutorial.pdf.
  2. Alonso, O. (2011). Perspectives on infrastructure for crowdsourcing. In M. Lease, V. Carvalho & E. Yilmaz (Eds.), Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), pp. 7–10.Google Scholar
  3. Alonso, O. (2013). Implementing crowdsourcing-based relevance experimentation: An industrial perspective. Information Retrieval, 16(2). doi:10.1007/s10791-012-9204-1.
  4. Alonso, O., & Lease, M. (2011a). Crowdsourcing 101: Putting the WSDM of crowds to work for you. In Proceedings of the fourth ACM international conference on web search and data mining (WSDM), pp. 1–2. Slides available online at http://ir.ischool.utexas.edu/wsdm2011_tutorial.pdf.
  5. Alonso, O., & Lease, M. (2011b). Crowdsourcing for information retrieval: Principles, methods, and applications. In Tutorial at the 34th annual ACM SIGIR conference, p. 1299, Beijing, China. Slides available online at http://www.slideshare.net/mattlease/crowdsourcing-for-information-retrieval-principles-methods-and-applications.
  6. Alonso, O., & Mizzaro, S. (2009). Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR workshop on the future of IR evaluation, pp. 15–16.Google Scholar
  7. Alonso, O., & Mizzaro, S. (2012). Using crowdsourcing for trec relevance assessment. Information Processing & Management.Google Scholar
  8. Alonso, O., Rose, D. E., & Stewart, B. (2008). Crowdsourcing for relevance evaluation. ACM SIGIR Forum, 42(2), 9–15.CrossRefGoogle Scholar
  9. Aslam, J. A., Pavlu, V., & Yilmaz, E. (2006). A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 541–548.Google Scholar
  10. Baio, A. (2008). The faces of mechanical turk. November 20. waxy.org/2008/11/the_faces_of_mechanical_turk.Google Scholar
  11. Carterette, B., Allan, J., & Sitaraman, R. (2006). Minimal test collections for retrieval evaluation. In Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp. 268–275.Google Scholar
  12. Carvalho, V., Lease, M., & Yilmaz, E. (2010). Crowdsourcing for search evaluation. ACM SIGIR Forum, 44(2), 17–22.CrossRefGoogle Scholar
  13. Chen, J. J., Menezes, N. J., Bradley, A. D., & North, T. (2011). Opportunities for crowdsourcing research on amazon mechanical turk. In CHI workshop on crowdsourcing and human computation.Google Scholar
  14. Cleverdon, C. (1997). The cranfield tests on index language devices. Readings in Information Retrieval, 47–59.Google Scholar
  15. Davis, J., Arderiu, J., Lin, H., Nevins, Z., Schuon, S., Gallo, O., et al. (2010). The HPU. In Computer vision and pattern recognition workshops (CVPRW), pp. 9–16.Google Scholar
  16. Eickhoff, C., Harris, C. G., Srinivasan, P., & de Vries, A. P. (2011). GEAnn—games for engaging annotations. In M. Lease, V. Hester, A. Sorokin & E. Yilmaz (Eds.), Proceedings of the ACM SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR 2011), p. 63, Beijing, China.Google Scholar
  17. Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012). Quality through flow and immersion: Gamifying crowdsourced relevance assessments. In Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval, pp. 871–880. ACM.Google Scholar
  18. Eickhoff, C., & de Vries, A. (2011). How crowdsourcable is your task? In M. Lease, V. Carvalho, & E. Yilmaz (Eds.), Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), pp. 11–14, Hong Kong, China. Received Most Innovative Paper Award.Google Scholar
  19. Eickhoff, C., & de Vries, A. P. (2013). Increasing cheat robustness of crowdsourcing tasks. Information Retrieval, 16(2). doi:10.1007/s10791-011-9181-9.
  20. Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon mechanical turk: Gold mine or coal mine? Computational Linguistics, 37(2), 413–420.CrossRefGoogle Scholar
  21. Grier, D. A. (2005). When computers were human, Vol. 316. Princeton: Princeton University Press.Google Scholar
  22. Harmanci, R. (2012). The googler who looked at the worst of the internet. BuzzFeed August 21. http://www.buzzfeed.com/reyhan/tech-confessional-the-googler-who-looks-at-the-wo.
  23. Howe, J. (2006). The rise of crowdsourcing. Wired Magazine 14(6), 1–4.MathSciNetGoogle Scholar
  24. Ipeirotis, P. (2010). Demographics of mechanical turk. Tech. Rep. CeDER-10-01, New York University.Google Scholar
  25. Irani, L., & Silberman, M. (2013). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceeding of the ACM SIGCHI conference on human factors in computing systems.Google Scholar
  26. Kazai, G. (2011). In search of quality in crowdsourcing for search engine evaluation. In European conference on information retrieval (ECIR), pp. 165–176.Google Scholar
  27. Kazai, G., Kamps, J., Koolen, M., & Milic-Frayling, N. (2011a). Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking. In SIGIR, pp. 205–214.Google Scholar
  28. Kazai, G., Kamps, J., & Milic-Frayling, N. (2011b). Worker types and personality traits in crowdsourcing relevance labels. In CIKM, pp. 1941–1944.Google Scholar
  29. Kazai, G., Kamps, J., & Milic-Frayling, N. (2012). The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In CIKM, pp. 2583–2586.Google Scholar
  30. Kazai, G., Kamps, J., & Milic-Frayling, N. (2013). An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information Retrieval, 16(2). doi:10.1007/s10791-012-9205-0.
  31. Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. In Proceedings of the ACM annual SIGCHI conference on Human factors in computing systems, pp. 453–456.Google Scholar
  32. Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., et al. (2013). The future of crowd work. In Proceedings of the ACM conference on computer supported cooperative work (CSCW), pp. 1301–1318.Google Scholar
  33. Kulkarni, A., Gutheim, P., Narula, P., Rolnitzky, D., Parikh, T., & Hartmann, B. (2012). Mobileworks: Designing for quality in a managed crowdsourcin architecture. IEEE Internet Computing, 16(5), 28.CrossRefGoogle Scholar
  34. Law, E., & Ahn, L. (2011). Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 5(3), 1–121.CrossRefGoogle Scholar
  35. Le, J., Edmonds, A., Hester, V., & Biewald, L. (2010). Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation, pp. 21–26.Google Scholar
  36. Lease, M., & Alonso, O. (2012). Crowdsourcing for search evaluation and social-algorithmic search. In Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’12. Slides available online at: http://www.slideshare.net/mattlease/crowdsourcing-for-search-evaluation-and-socialalgorithmic-search.
  37. Lease, M., Carvalho, V., & Yilmaz, E. (eds.). (2010). Proceedings of the ACM SIGIR 2010 workshop on crowdsourcing for search evaluation (CSE 2010). Geneva, Switzerland. Available online at http://ir.ischool.utexas.edu/cse2010.
  38. Lease, M., Carvalho, V., & Yilmaz, E. (2011a). Crowdsourcing for search and data mining. ACM SIGIR Forum, 45(1), 18–24.CrossRefGoogle Scholar
  39. Lease, M., Carvalho, V., & Yilmaz, E. (eds.). (2011b). Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), Hong Kong, China. Available online at ir.ischool.utexas.edu/csdm2011/proceedings.html.Google Scholar
  40. Lease, M., & Kazai, G. (2011a). Overview of the TREC 2011 crowdsourcing track (conference notebook). In 20th text retrieval conference (TREC).Google Scholar
  41. Lease, M., & Yilmaz, E. (2011b). Crowdsourcing for information retrieval. ACM SIGIR Forum, 45(2), 66–75.CrossRefGoogle Scholar
  42. Lease, M., Yilmaz, E., Sorokin, A., & Hester, V. (eds.). (2011c). Proceedings of the 2nd ACM SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR). Beijing, China. Available online at http://sites.google.com/site/cir2011ws/proceedings.
  43. McCreadie, R., Macdonald, C., & Ounis, I. (2011). Crowdsourcing blog track top news judgments at TREC. In M. Lease, V. Carvalho & E. Yilmaz (Eds.), Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), pp. 23–26, Hong Kong, China.Google Scholar
  44. McCreadie, R., Macdonald, C., & Ounis, I. (2012). Crowdterrier: Automatic crowdsourced relevance assessments with terrier. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pp. 1005–1005. ACM.Google Scholar
  45. McCreadie, R., Macdonald, C., & Ounis, I. (2013). Identifying top news using crowdsourcing. Information Retrieval, 16(2). doi:10.1007/s10791-012-9186-z.
  46. Mitchell, S. (2010). Inside the online sweatshops. PC Pro Magazine. August 6.http://www.pcpro.co.uk/features/360127/inside-the-online-sweatshops.
  47. Munro, R. (2013). Crowdsourcing and the crisis-affected community lessons learned and looking forward from mission 4636. Information Retrieval, 16(2). doi:10.1007/s10791-012-9203-2.
  48. Narula, P., Gutheim, P., Rolnitzky, D., Kulkarni, A., & Hartmann, B. (2011). Mobileworks: A mobile crowdsourcing platform for workers at the bottom of the pyramid. In AAAI human computation workshop.Google Scholar
  49. Oleson, D., Sorokin, A., Laughlin, G., Hester, V., Le, J., & Biewald, L. (2011). Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. In AAAI workshop on human computation.Google Scholar
  50. Ounis, I., & Soboroff, I. (2011). Overview of the TREC 2010 blog track. In Proceedings of the 19th Text REtrieval Conference (TREC).Google Scholar
  51. Paritosh, P., Ipeirotis, P., Cooper, M., & Suri, S. (2011). The computer is the new sewing machine: Benefits and perils of crowdsourcing. In Proceedings of the 20th international conference companion on world wide web, pp. 325–326. ACM.Google Scholar
  52. Quinn, A. J., & Bederson, B. B. (2011). Human computation: A survey and taxonomy of a growing field. In 2011 Annual ACM SIGCHI conference on Human factors in computing systems, pp. 1403–1412.Google Scholar
  53. Richard, M. C., McCreadie, C. M., & Ounis, I. (2010). Crowdsourcing a news query classification dataset. In M. Lease, V. Carvalho & E. Yilmaz (Eds.), Proceedings of the ACM SIGIR 2010 workshop on crowdsourcing for search evaluation (CSE 2010), pp. 31–38. Geneva, Switzerland.Google Scholar
  54. Ross, J., Irani, L., Silberman, M., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers?: Shifting demographics in mechanical turk. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, pp. 2863–2872. ACM.Google Scholar
  55. Silberman, M., Irani, L., & Ross, J. (2010). Ethics and tactics of professional crowdwork. XRDS: Crossroads, The ACM Magazine for Students, 17(2), 39–43.CrossRefGoogle Scholar
  56. Smucker, M., Kazai, G., & Lease, M. (2013). Overview of the TREC 2012 crowdsourcing Track. In Proceedings of the 21st NIST text retrieval conference (TREC).Google Scholar
  57. Snow, R., O’Connor, B., Jurafsky, D., & Ng, A.Y. (2008). Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In: Proceedings of the conference on empirical methods in natural language processing, pp. 254–263. Association for Computational Linguistics.Google Scholar
  58. Surowiecki, J. (2005). The wisdom of crowds. Anchor.Google Scholar
  59. Vliegendhart, R., Larson, M., Kofler, C., Eickhoff, C., & Pouwelse, J. (2011). Investigating factors influencing crowdsourcing tasks with high imaginative load. In M. Lease, V. Carvalho & E. Yilmaz (Eds.), Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), pp. 27–30.Google Scholar
  60. Vuurens, J., Vries, A. P. D., & Eickhoff, C. (2011). How much spam can you take? An analysis of crowdsourcing results to increase accuracy. In M. Lease, V. Hester, A. Sorokin & E. Yilmaz (Eds.), Proceedings of the ACM SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR 2011), pp. 48–55.Google Scholar
  61. Vuurens, J.B., & de Vries, A. P. (2012). Obtaining high-quality relevance judgments using crowdsourcing. IEEE Internet Computing 16(5), 20–27. http://doi.ieeecomputersociety.org/10.1109/MIC.2012.71.Google Scholar
  62. Yan, T., Kumar, V., & Ganesan, D. (2010). CrowdSearch: Exploiting crowds for accurate real-time image search on mobile phones. In Proceedings of the 8th international conference on mobile systems, applications, and services (MOBISYS), pp. 77–90. ACM.Google Scholar
  63. Yilmaz, E., Kanoulas, E., & Aslam, J. A. (2008). A simple and efficient sampling method for estimating AP and NDCG. In Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval, pp. 603–610.Google Scholar
  64. Zuccon, G., Leelanupab, T., Whiting, S., Jose, J., & Azzopardi, L. (2011). Crowdsourcing interactions—a proposal for capturing user interactions through crowdsourcing. In M. Lease, V. Carvalho & E. Yilmaz (Eds.), Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), pp. 35–38.Google Scholar
  65. Zuccon, G., Leelanupab, T., Whiting, S., Yilmaz, E., Jose, J. M., & Azzopardi, L. (2013). Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems. Information Retrieval, 16(2). doi:10.1007/s10791-012-9206-z.

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.School of InformationUniversity of Texas at AustinAustinUSA
  2. 2.Microsoft ResearchCambridgeUK
  3. 3.Koc UniversityIstanbulTurkey

Personalised recommendations