Skip to main content
SpringerLink
Log in

Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd

  • Conference paper
  • First Online:
Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10264))

Abstract

Crowdsourcing solutions are increasingly being adopted across a variety of domains these days. An important consequence of the flourishing crowdsourcing markets is that experiments which were traditionally carried out in laboratories on a much smaller scale can now tap into the immense potential of online labor. Researchers in different fields have shown considerable interest in attempting to carry out priorly constrained lab experiments in the crowd. In this chapter, we reflect on the key factors to consider while transitioning from controlled laboratory experiments to large scale experiments in the crowd.

The original version of this chapter was revised. The affiliation of the third author was corrected. The erratum to this chapter is available at https://doi.org/10.1007/978-3-319-66435-4_8

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.wired.com/2006/06/crowds/ last accessed 14 Jun 2017.

  2. 2.

    http://crowdflower.com/ last accessed 14 Jun 2017.

  3. 3.

    https://www.mturk.com/ last accessed 14 Jun 2017.

  4. 4.

    http://www.crowdflower.com/ last accessed 14 Jun 2017.

  5. 5.

    https://www.upwork.com/ last accessed 14 Jun 2017.

References

  1. Anderson, J.R., Matessa, M., Lebiere, C.: ACT-R: a theory of higher level cognition and its relation to visual attention. Hum. Comput. Interact. 12(4), 439–462 (1997)

    Article  Google Scholar 

  2. Campbell, D.J.: Task complexity: a review and analysis. Acad. Manag. Rev. 13(1), 40–52 (1988)

    Google Scholar 

  3. Cheng, J., Teevan, J., Bernstein, M.S.: Measuring crowdsourcing effort with error-time curves. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1365–1374. ACM (2015)

    Google Scholar 

  4. Chung, D.H.S., Archambault, D., Borgo, R., Edwards, D.J., Laramee, R.S., Chen, M.: How ordered is it? On the perceptual orderability of visual channels. Comput. Graph. Forum 35(3), 131–140 (2016). (Proc. of EuroVis 2016)

    Article  Google Scholar 

  5. Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., Singh, M.: How well do line drawings depict shape? ACM Trans. Graph. 28(3), 1–9 (2009)

    Article  Google Scholar 

  6. Cozby, P.: Asking people about themselves: survey research. In: Methods in Behavioral Research, 7th edn., pp. 103–124. Mayfield Publishing Company, Mountain View (2001)

    Google Scholar 

  7. Crump, M.J., McDonnell, J.V., Gureckis, T.M.: Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PloS one 8(3), e57410 (2013)

    Article  Google Scholar 

  8. Difallah, D.E., Catasta, M., Demartini, G., Cudré-Mauroux, P.: Scaling-up the crowd: micro-task pricing schemes for worker retention and latency improvement. In: Second AAAI Conference on Human Computation and Crowdsourcing (2014)

    Google Scholar 

  9. Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Mechanical cheat: spamming schemes and adversarial techniques on crowdsourcing platforms. In: CrowdSearch, pp. 26–30. Citeseer (2012)

    Google Scholar 

  10. Dow, S., Kulkarni, A., Klemmer, S., Hartmann, B.: Shepherding the crowd yields better work. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, pp. 1013–1022. ACM (2012)

    Google Scholar 

  11. Eickhoff, C., de Vries, A.P.: Increasing cheat robustness of crowdsourcing tasks. Inf. Retr. 16(2), 121–137 (2013)

    Article  Google Scholar 

  12. Feyisetan, O., Luczak-Roesch, M., Simperl, E., Tinati, R., Shadbolt, N.: Towards hybrid NER: a study of content and crowdsourcing-related performance factors. In: Gandon, F., Sabou, M., Sack, H., d’Amato, C., Cudré-Mauroux, P., Zimmermann, A. (eds.) ESWC 2015. LNCS, vol. 9088, pp. 525–540. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18818-8_32

    Chapter  Google Scholar 

  13. Fikkert, W., D’Ambros, M., Bierz, T., Jankun-Kelly, T.J.: Interacting with visualizations. In: Kerren, A., Ebert, A., Meyer, J. (eds.) Human-Centered Visualization Environments. LNCS, vol. 4417, pp. 77–162. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-71949-6_3

    Chapter  Google Scholar 

  14. Fu, W.T., Pirolli, P.: SNIF-ACT: a cognitive model of user navigation on the world wide web. Hum. Comput. Interact. 22(4), 355–412 (2007)

    Google Scholar 

  15. Gadiraju, U.: Crystal clear or very vague? Effects of task clarity in the microtask crowdsourcing ecosystem. In: 1st International Workshop on Weaving Relations of Trust in Crowd Work: Transparency and Reputation Across Platforms, Co-located With the 8th International ACM Web Science Conference 2016, Hannover (2016)

    Google Scholar 

  16. Gadiraju, U., Dietze, S.: Improving learning through achievement priming in crowdsourced information finding microtasks. In: Proceedings of ACM LAK Conference. ACM (2017, to appear)

    Google Scholar 

  17. Gadiraju, U., Fetahu, B., Kawase, R.: Training workers for improving performance in crowdsourcing microtasks. In: Conole, G., Klobučar, T., Rensing, C., Konert, J., Lavoué, É. (eds.) EC-TEL 2015. LNCS, vol. 9307, pp. 100–114. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24258-3_8

    Chapter  Google Scholar 

  18. Gadiraju, U., Kawase, R., Dietze, S.: A taxonomy of microtasks on the web. In: Proceedings of the 25th ACM Conference on Hypertext and Social Media, pp. 218–223. ACM (2014)

    Google Scholar 

  19. Gadiraju, U., Kawase, R., Dietze, S., Demartini, G.: Understanding malicious behavior in crowdsourcing platforms: the case of online surveys. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI 2015), Seoul, 18–23 April 2015, pp. 1631–1640 (2015)

    Google Scholar 

  20. Gadiraju, U., Siehndel, P., Fetahu, B., Kawase, R.: Breaking bad: understanding behavior of crowd workers in categorization microtasks. In: Proceedings of the 26th ACM Conference on Hypertext & Social Media, pp. 33–38. ACM (2015)

    Google Scholar 

  21. Gardlo, B., Egger, S., Seufert, M., Schatz, R.: Crowdsourcing 2.0: enhancing execution speed and reliability of web-based QoE testing. In: Proceedings of the IEEE International Conference on Communications (ICC), pp. 1070–1075 (2014)

    Google Scholar 

  22. Goncalves, J., Ferreira, D., Hosio, S., Liu, Y., Rogstadius, J., Kukka, H., Kostakos, V.: Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 753–762. ACM (2013)

    Google Scholar 

  23. Hanhart, P., Korshunov, P., Ebrahimi, T.: Crowd-based quality assessment of multiview video plus depth coding. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 743–747. IEEE (2014)

    Google Scholar 

  24. Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the 28th International Conference on Human Factors in Computing Systems (CHI 2010), Atlanta, 10–15 April 2010, pp. 203–212 (2010)

    Google Scholar 

  25. Heinzelman, J., Waters, C.: Crowdsourcing crisis information in disaster-affected Haiti. US Institute of Peace (2010)

    Google Scholar 

  26. Horton, J.J., Rand, D.G., Zeckhauser, R.J.: The online laboratory: conducting experiments in a real labor market. Exp. Econ. 14(3), 399–425 (2011)

    Article  Google Scholar 

  27. Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., Tran-Gia, P.: Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Trans. Multimed. 16(2), 541–558 (2014)

    Article  Google Scholar 

  28. Hoßfeld, T., Tran-Gia, P., Vucovic, M.: Crowdsourcing: from theory to practice and long-term perspectives (Dagstuhl Seminar 13361). Dagstuhl Rep. 3(9), 1–33 (2013). http://drops.dagstuhl.de/opus/volltexte/2013/4354

    Google Scholar 

  29. ITU-T Rec. P.805: Subjective evaluation of conversational quality. International Telecommunication Union, Geneva (2007)

    Google Scholar 

  30. Ipeirotis, P.G.: Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads ACM Mag. Stud. 17(2), 16–21 (2010)

    Article  Google Scholar 

  31. Ipeirotis, P.G.: Demographics of Mechanical Turk (2010)

    Google Scholar 

  32. Isenberg, P., Elmqvist, N., Scholtz, J., Cernea, D., Ma, K.L., Hagen, H.: Collaborative visualization: definition, challenges, and research agenda. Inf. Vis. 10(4), 310–326 (2011)

    Article  Google Scholar 

  33. Khatib, F., Cooper, S., Tyka, M.D., Xu, K., Makedon, I., Popović, Z., Baker, D., Players, F.: Algorithm discovery by protein folding game players. Proc. Natl. Acad. Sci. 108(47), 18949–18953 (2011)

    Article  Google Scholar 

  34. Khatib, F., DiMaio, F., Cooper, S., Kazmierczyk, M., Gilski, M., Krzywda, S., Zabranska, H., Pichova, I., Thompson, J., Popović, Z., et al.: Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat. Struct. Mol. Biol. 18(10), 1175–1177 (2011)

    Article  Google Scholar 

  35. Lebreton, P.R., Mäki, T., Skodras, E., Hupont, I., Hirth, M.: Bridging the gap between eye tracking and crowdsourcing. In: Human Vision and Electronic Imaging XX, San Francisco, 9–12 February 2015, p. 93940W (2015)

    Google Scholar 

  36. Marshall, C.C., Shipman, F.M.: Experiences surveying the crowd: reflections on methods, participation, and reliability. In: Proceedings of the 5th Annual ACM Web Science Conference, pp. 234–243. ACM (2013)

    Google Scholar 

  37. Mason, W., Suri, S.: Conducting behavioral research on Amazons Mechanical Turk. Behav. Res. Methods 44(1), 1–23 (2012)

    Article  Google Scholar 

  38. McCrae, J., Mitra, N.J., Singh, K.: Surface perception of planar abstractions. ACM Trans. Appl. Percept. 10(3), 14: 1–14: 20 (2013)

    Article  Google Scholar 

  39. Okoe, M., Jianu, R.: GraphUnit: evaluating interactive graph visualizations using crowdsourcing. Comput. Graph. Forum 34(3), 451–460 (2015)

    Article  Google Scholar 

  40. Oleson, D., Sorokin, A., Laughlin, G., Hester, V., Le, J., Biewald, L.: Programmatic gold: targeted and scalable quality assurance in crowdsourcing. In: Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence (WS-11-11). AAAI (2011)

    Google Scholar 

  41. Paolacci, G., Chandler, J., Ipeirotis, P.G.: Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 5(5), 411–419 (2010)

    Google Scholar 

  42. Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: Proceedings of International Conference on Intelligence Analysis, vol. 5, pp. 2–4 (2005)

    Google Scholar 

  43. Pylyshyn, Z.W.: Things and Places: How the Mind Connects with the World. MIT Press, Cambridge (2007)

    Google Scholar 

  44. Rand, D.G.: The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. J. Theor. Biol. 299, 172–179 (2012)

    Article  MathSciNet  Google Scholar 

  45. Rokicki, M., Chelaru, S., Zerr, S., Siersdorfer, S.: Competitive game designs for improving the cost effectiveness of crowdsourcing. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, pp. 1469–1478. ACM (2014)

    Google Scholar 

  46. Rokicki, M., Zerr, S., Siersdorfer, S.: Groupsourcing: team competition designs for crowdsourcing. In: Proceedings of the 24th International Conference on World Wide Web, pp. 906–915. International World Wide Web Conferences Steering Committee (2015)

    Google Scholar 

  47. Salehi, N., Irani, L.C., Bernstein, M.S., Alkhatib, A., Ogbe, E., Milland, K., et al.: We are dynamo: overcoming stalling and friction in collective action for crowd workers. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1621–1630. ACM (2015)

    Google Scholar 

  48. Tetlock, P.E., Mellers, B.A., Rohrbaugh, N., Chen, E.: Forecasting tournaments tools for increasing transparency and improving the quality of debate. Curr. Dir. Psychol. Sci. 23(4), 290–295 (2014)

    Article  Google Scholar 

  49. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 319–326. ACM (2004)

    Google Scholar 

  50. Weber, L., Silverman, R.E.: On-demand workers: we are not robots. Wall Str. J. 7 (2015)

    Google Scholar 

  51. Williamson, V.: On the ethics of crowdsourced research. PS Political Sci. Politics 49(01), 77–81 (2016)

    Article  Google Scholar 

  52. Yang, J., Redi, J., DeMartini, G., Bozzon, A.: Modeling task complexity in crowdsourcing. In: Proceedings of the Fourth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016), pp. 249–258. AAAI (2016)

    Google Scholar 

Download references

Acknowledgment

We would like to thank Dagstuhl for facilitating the seminar (titled, ‘Evaluation in the Crowd: Crowdsourcing and Human-Centred Experiments’) that brought about this collaboration. Part of this work (Sect. 4) was supported by the German Research Foundation (DFG) within project A05 of SFB/Transregio 161. We also thank Andrea Mauri and Christian Keimel for their valuable contributions and feedback during discussions.

Author information

Authors and Affiliations

  1. Leibniz Universität Hannover, Hannover, Germany

    Ujwal Gadiraju

  2. TU Berlin, Berlin, Germany

    Sebastian Möller

  3. Algorithms and Complexity Group, TU Wien, Vienna, Austria

    Martin Nöllenburg

  4. University of Konstanz, Konstanz, Germany

    Dietmar Saupe

  5. Austrian Institute of Technology, Vienna, Austria

    Sebastian Egger-Lampl

  6. Swansea University, Swansea, UK

    Daniel Archambault

  7. Simon Fraser University, Burnaby, Canada

    Brian Fisher

Authors
  1. Ujwal Gadiraju
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Möller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martin Nöllenburg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dietmar Saupe
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sebastian Egger-Lampl
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel Archambault
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Brian Fisher
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ujwal Gadiraju .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Swansea University, Swansea, United Kingdom

    Daniel Archambault

  2. University of Glasgow, Glasgow, United Kingdom

    Helen Purchase

  3. Modellierung adaptiver Systeme, Universität Duisburg-Essen, Essen, Germany

    Tobias Hoßfeld

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Gadiraju, U. et al. (2017). Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd. In: Archambault, D., Purchase, H., Hoßfeld, T. (eds) Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Lecture Notes in Computer Science(), vol 10264. Springer, Cham. https://doi.org/10.1007/978-3-319-66435-4_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-66435-4_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-66434-7

  • Online ISBN: 978-3-319-66435-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation