Advertisement

Computer Supported Cooperative Work (CSCW)

, Volume 27, Issue 3–6, pp 1275–1324 | Cite as

Rating Working Conditions on Digital Labor Platforms

  • Ellie Harmon
  • M. Six Silberman
Article

Abstract

The relations between technology, work organization, worker power, workers’ rights, and workers’ experience of work have long been central concerns of CSCW. European CSCW research, especially, has a tradition of close collaboration with workers and trade unionists in which researchers aim to develop technologies and work processes that increase workplace democracy. This paper contributes a practitioner perspective on this theme in a new context: the (sometimes global) labor markets enabled by digital labor platforms. Specifically, the paper describes a method for rating working conditions on digital labor platforms (e.g., Amazon Mechanical Turk, Uber) developed within a trade union setting. Preliminary results have been made public on a website that is referred to by workers, platform operators, journalists, researchers, and policy makers. This paper describes this technical project in the context of broader cross-sectoral efforts to safeguard worker rights and build worker power in digital labor platforms. Not a traditional research paper, this article instead takes the form of a case study documenting the process of incorporating a human-centered computing perspective into contemporary trade union activities and communicating a practitioner’s perspective on how CSCW research and computational artifacts can come to matter outside of the academy. The paper shows how practical applications can benefit from the work of CSCW researchers, while illustrating some practical constraints of the trade union context. The paper also offers some practical contributions for researchers studying digital platform workers’ experiences and rights: the artifacts and processes developed in the course of the work.

Key words

crowd work crowdsourcing digital labor digitalization gig economy labor unions platform-based work policy rating schemes surveys workers’ rights working conditions automated management automated management systems future of work 

Notes

Acknowledgements

The work described in this paper (but not the preparation of the paper) was funded by IG Metall (the German Metalworkers’ Union), the Austrian Chamber of Labor (Arbeiterkammer), and Unionen, but the paper was not approved or reviewed by any of these organizations, nor does it reflect any official organizational position. The authors gratefully acknowledge ongoing discussions with Christiane Benner, Vanessa Barth, and Robert Fuss at IG Metall, Sylvia Kuba at the Austrian Chamber of Labor, Karin Zimmermann at the Austrian Confederation of Trade Unions, Fredrik Söderqvist and Carin Hallerström at Unionen, Mattias Beijmo at DUMA, Janine Berg at the International Labour Office, Valerio De Stefano, Mark Graham, Vili Lehdonvirta, Jamie Woodcock, Lilly Irani, Rochelle LaPlante, Benjamin Herr, and Dawn Gearhart. The paper has benefited from suggestions from Erin Goodling, two anonymous reviewers, the editors of this special issue of JCSCW, and the JCSCW editors. The term “automated management systems” is owed to Janine Berg.

References

  1. Adam, D., M. Bremermann, J. D. López, et al. (2016). Digitalisation and working life: lessons from the Uber cases around Europe. EurWORK | European Observatory of Working Life, 25 January 2016. https://www.eurofound.europa.eu/observatories/eurwork/articles/working-conditions-law-and-regulation-business/digitalisation-and-working-life-lessons-from-the-uber-cases-around-europe.
  2. Amazon Mechanical Turk (2014). Participation Agreement. 2 December 2014. http://mturk.com/mturk/conditionsofuse. Accessed 13 May 2017.
  3. Austrian Chamber of Labour, Austrian Trade Union Confederation, Danish Union of Commercial and Clerical Workers, et al. (2016). Frankfurt Paper on Platform Based Work (Report). Frankfurt: IG Metall (German Metalworkers’ Union). http://tinyurl.com/hdrljk2. Accessed 1 January 2018.
  4. Bederson, B. B., and A. J. Quinn (2011). Web Workers Unite! Addressing Challenges of Online Laborers. In CHI EA ‘11. CHI ‘11 Extended Abstracts on Human Factors in Computing Systems, Vancouver, BC, Canada, 07–12 May 2011. New York: ACM, pp. 97–106.  https://doi.org/10.1145/1979742.1979606.Google Scholar
  5. Benner, C. (ed.) (2015). Crowdwork — zurück in die Zukunft? Perspektiven digitale Arbeit. Frankfurt: Bund Verlag.Google Scholar
  6. Benson, A., A. J. Sojourner, and A. Umyarov (2015). The value of employer reputation in the absence of contract enforcement: A randomized experiment. Presented at the Allied Social Science Associations Annual Meeting, Boston, MA, USA, 3–5 January 2015.Google Scholar
  7. Berg, J. (2016). Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law and Policy Journal, vol. 37, no. 3, pp. 543–576.Google Scholar
  8. Bjerknes, G., P. Ehn, M. Kyng, et al. (eds) (1987). Computers and Democracy: A Scandinavian Challenge. England: Avebury.Google Scholar
  9. Bjerknes, G., and T. Bratteteig (1995). User participation and democracy: A discussion of Scandinavian research on system development. Scandinavian Journal of Information Systems, vol. 7, no. 1, pp. 73–98.Google Scholar
  10. Bohannon, J. (2016). Psychologists grow increasingly dependent on online research subjects. Science Magazine, 7 June 2016. http://www.sciencemag.org/news/2016/06/psychologists-grow-increasingly-dependent-online-research-subjects. Accessed 1 January 2018.
  11. Bradshaw, T. (2017). Self-driving cars prove to be labour-intensive for humans. Financial Times, 9 July 2017.Google Scholar
  12. Brian (@xyderias), Kristy Milland (@TurkerNational), and Rochelle (@Rochelle) (2017). MTurk requester tip: if your attention check begins with “This is an attention check,” it isn’t. [Tweet]. 7 April 2017, 7 April 2017. https://twitter.com/xyderias/status/857239009633812481. Accessed 3 January 2018.
  13. Callison-Burch, C. (2014). Crowd-Workers: Aggregating Information Across Turkers to Help Them Find Higher Paying Work. In Second AAAI Conference on Human Computation and Crowdsourcing, Pittsburgh, PA, USA, 02–04 November 2014.Palo Alto: AAAI Publications. https://www.aaai.org/ocs/index.php/HCOMP/HCOMP14/paper/view/9067. Accessed 1 January 2018.Google Scholar
  14. Chandler, J., P. Mueller, and G. Paolacci (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, vol. 46, no. 1, pp. 112–130.  https://doi.org/10.3758/s13428-013-0365-7.CrossRefGoogle Scholar
  15. Chrisafis, A. (2016). France hit by day of protest as security forces fire teargas at taxi strike. The Guardian, 26 January 2016. http://www.theguardian.com/world/2016/jan/26/french-taxi-drivers-block-paris-roads-in-uber-protest. Accessed 1 January 2018.
  16. Court of Justice of the European Union (2017). According to Advocate General Szpunar, the Uber electronic platform, whilst innovative, falls within the field of transport: Uber can thus be required to obtain the necessary licences and authorisations under national law (Press Release, no. 50/17). Luxembourg. https://curia.europa.eu/jcms/upload/docs/application/pdf/2017-05/cp170050en.pdf.
  17. De Stefano, V. (2016a). Introduction: Crowdsourcing, the Gig-Economy and the Law. Comparative Labor Law and Policy Journal, vol. 37, no. 3.Google Scholar
  18. De Stefano, V. (2016b). The rise of the “just-in-time workforce”: on-demand work, crowd work, and labour protection in the “gig economy”. Comparative Labor Law and Policy Journal, vol. 37, no. 3.Google Scholar
  19. Dølvik, J. E., and K. Jesnes (2017). Nordic Labour Markets and the Sharing Economy — Report from a Pilot Project (Report). Copenhagen: Nordic Council of Ministers.CrossRefGoogle Scholar
  20. Dow, S., A. Kulkarni, S. Klemmer, et al. (2012). Shepherding the Crowd Yields Better Work. In CSCW ‘12. Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, Seattle, Washington, USA, 11–15 February 2012. New York: ACM, pp. 1013–1022.  https://doi.org/10.1145/2145204.2145355.Google Scholar
  21. Downs, J. S., M. B. Holbrook, S. Sheng, et al. (2010). Are Your Participants Gaming the System?: Screening Mechanical Turk Workers. In CHI ‘10. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, Georgia, USA, 10–15 April 2010. New York: ACM, pp. 2399–2402.  https://doi.org/10.1145/1753326.1753688.Google Scholar
  22. Drahokoupil, J., and M. Jepsen (2017). The digital economy and its implications for labour. 1. The platform economy. Transfer: European Review of Labour and Research, vol. 23, no. 2, pp. 103–107.  https://doi.org/10.1177/1024258917701380.CrossRefGoogle Scholar
  23. Ehn, P. (1990). Work-Oriented Design of Computer Artifacts. Hillsdale: L. Erlbaum Associates.Google Scholar
  24. European Commission (2016). A European agenda for the collaborative economy (Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, no. COM (2016) 356 final). Brussels: European Commission.Google Scholar
  25. European Parliament (2017). European agenda for the collaborative economy (INI - Own-initiative procedure, no. 2017/2003(INI)). http://www.europarl.europa.eu/oeil/popups/ficheprocedure.do?lang=en&reference=2017/2003(INI).
  26. Forde, C., M. Stuart, S. Joyce, et al. (2017). The Social Protection of Workers in the Platform Economy (Study for the EMPL Committee, no. IP/A/EMPL/2016-11 PE 614.184). Brussels: European Parliament | Directorate-General for Internal Policies | Policy Department A: Economic and Scientific Policy. http://www.europarl.europa.eu/RegData/etudes/STUD/2017/614184/IPOL_STU%282017%29614184_EN.pdf. Accessed 1 January 2018.
  27. Gadiraju, U., B. Fetahu, and C. Hube (2016). Crystal clear or very vague? Effects of task clarity in the microtask crowdsourcing ecosystem. In 1st International Workshop on Weaving Relations of Trust in Crowd Work: Transparency and Reputation Across Platforms, Co-located with the 8th International ACM Web Science Conference, Hannover, Germany, 22–25 May 2016.Google Scholar
  28. Gaikwad, S. N., D. Morina, R. Nistala, et al. (2015). Daemo: A Self-Governed Crowdsourcing Marketplace. In UIST ‘15 Adjunct. Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, Daegu, Kyungpook, Republic of Korea, 08–11 November 2015. New York: ACM, pp. 101–102.  https://doi.org/10.1145/2815585.2815739.
  29. Gaikwad, S. N. S., D. Morina, A. Ginzberg, et al. (2016). Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms. In UIST ‘16. Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, 16–19 October 2016. New York, NY, USA: ACM, pp. 625–637.  https://doi.org/10.1145/2984511.2984542.
  30. Gaikwad, S. N., M. E. Whiting, D. Gamage, et al. (2017). The Daemo Crowdsourcing Marketplace. In CSCW ‘17 Companion. Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, Oregon, USA, 25 February - 01 March 2017. New York: ACM, pp. 1–4.  https://doi.org/10.1145/3022198.3023270.
  31. Garden City Group (n.d.). Douglas O’Connor v. Uber Technologies, Inc. [Lawsuit Website]. http://www.uberlitigation.com/. Accessed 2 January 2018.
  32. Gleibs, I. H. (2017). Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places. Behavior Research Methods, vol. 49, no. 4, pp. 1333–1342.  https://doi.org/10.3758/s13428-016-0789-y.CrossRefGoogle Scholar
  33. Gordon, C. and R. Eisenbrey (2012). As unions decline, inequality rises. Economic Policy Institute Publications. 6 June 2012Economic Policy Institute Publications, 6 June 2012. https://www.epi.org/publication/unions-decline-inequality-rises/. Accessed 8 April 2018.
  34. Graham, M., I. Hjorth, and V. Lehdonvirta (2017). Digital labour and development: impacts of global digital labour platforms and the gig economy on worker livelihoods. Transfer: European Review of Labour and Research, vol. 23, no. 2, pp. 135–162.  https://doi.org/10.1177/1024258916687250.CrossRefGoogle Scholar
  35. Greenbaum, J. (1988). In Search of Cooperation: An Historical Analysis of Work Organization and Management Strategies. In CSCW ‘88. Proceedings of the 1988 ACM Conference on Computer-supported Cooperative Work, Portland, Oregon, USA, 26–28 September 1988. New York: ACM, pp. 102–114.  https://doi.org/10.1145/62266.62275.Google Scholar
  36. Greenbaum, J. (1996). Back to Labor: Returning to Labor Process Discussions in the Study of Work. In CSCW ‘96. Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, Boston, Massachusetts, USA, 16–20 November 1996. New York: ACM, pp. 229–237.  https://doi.org/10.1145/240080.240259.Google Scholar
  37. Greenbaum, J., and M. Kyng (Eds.) (1991). Design at Work: Cooperative Design of Computer Systems. Hillsdale: Lawrence Erlbaum Associates, Inc., Publishers.Google Scholar
  38. Gupta, N., D. Martin, B. V. Hanrahan; et al. (2014). Turk-Life in India. In GROUP ‘14. Proceedings of the 18th International Conference on Supporting Group Work, Sanibel Island, Florida, USA, 09–12 November 2014. New York: ACM, pp. 1–11.  https://doi.org/10.1145/2660398.2660403.Google Scholar
  39. Hanrahan, B. V., J. K. Willamowski, S. Swaminathan; et al. (2015). TurkBench: Rendering the Market for Turkers. In CHI ‘15. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, April 18–23, 2015. New York: ACM, pp. 1613–1616.  https://doi.org/10.1145/2702123.2702279.Google Scholar
  40. Hara, K., A. Adams, K. Milland, et al. (2018). A Data-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk. In CHI ‘18. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montréal, QC, Canada, 21–26 April 2018. New York: ACM.Google Scholar
  41. Haug, M. C. (2017). Fast, Cheap, and Unethical? The Interplay of Morality and Methodology in Crowdsourced Survey Research. Review of Philosophy and Psychology, pp. 1–17.  https://doi.org/10.1007/s13164-017-0374-z.
  42. Hauser, D. J., and N. Schwarz (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, vol. 48, no. 1, pp. 400–407.  https://doi.org/10.3758/s13428-015-0578-z.CrossRefGoogle Scholar
  43. Herr, B. (2017). Bericht Plattformunternehmen Foodora Wien (Internal Report). Austrian Chamber of Labour.Google Scholar
  44. Hickey, S. (2016). Uber tribunal judges criticise “fictions” and “twisted language.” The Guardian, 28 October 2016. http://www.theguardian.com/technology/2016/oct/28/uber-tribunal-judges-fictions-twisted-language-appeal. Accessed 2 January 2018.
  45. Houle, C. (2009). Inequality and Democracy: Why Inequality Harms Consolidation but Does Not Affect Democratization. World Politics, vol. 61, no. 4, pp. 589–622.  https://doi.org/10.1017/S0043887109990074.CrossRefGoogle Scholar
  46. Huws, U., N. H. Spencer, and S. Joyce (2016). Crowd Work in Europe: Preliminary Results from a Survey in the UK, Sweden, Germany, Austria and the Netherlands. (Research Report). Brussels: Foundation for European Progressive Studies. http://www.feps-europe.eu/assets/39aad271-85ff-457c-8b23-b30d82bb808f/crowd-work-in-europe-draft-report-last-versionpdf.pdf.
  47. International Labour Organization (2017). Inception Report for the Global Commission on the Future of Work (Inception Report, no. ISBN 978-92-2-131372-4). Geneva: International Labour Office.Google Scholar
  48. Ipeirotis, P. G. (2010). Demographics of Mechanical Turk (Working Paper, no. CeDER-10-01). New York University | Center for Digital Economy Research. http://hdl.handle.net/2451/29585. Accessed 2 January 2018.
  49. Ipeirotis, P. (2012). Why I love crowdsourcing (the concept) and hate crowdsourcing (the term). A Computer Scientist in a Business School. 13 November 2012A Computer Scientist in a Business School, 13 November 2012. http://www.behind-the-enemy-lines.com/2012/11/why-i-love-crowdsourcing-concept-and.html. Accessed 3 January 2018.
  50. Ipeirotis, P. G., F. Provost, and J. Wang (2010). Quality Management on Amazon Mechanical Turk. In HCOMP ‘10. Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, USA, 25 July 2010. New York: ACM, pp. 64–67.  https://doi.org/10.1145/1837885.1837906.Google Scholar
  51. Irani, L. (2015). The cultural work of microwork. New Media & Society, vol. 17, no. 5, pp. 720–739.  https://doi.org/10.1177/1461444813511926.CrossRefGoogle Scholar
  52. Irani, L. C., and M. S. Silberman (2013). Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In CHI ‘13. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, 27 April - 02 May 2013. New York: ACM, pp. 611–620.  https://doi.org/10.1145/2470654.2470742.Google Scholar
  53. Irani, L. C., and M. S. Silberman (2014). From critical design to critical infrastructure: lessons from turkopticon. interactions, vol. 21, no. 4, pp. 32–35.CrossRefGoogle Scholar
  54. Irani, L. C., and M. S. Silberman (2016). Stories We Tell About Labor: Turkopticon and the Trouble with “Design”. In CHI ‘16. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, California, USA, 07–12 May 2016. New York: ACM, pp. 4573–4586.  https://doi.org/10.1145/2858036.2858592.Google Scholar
  55. Jaumotte, F., and C. O. Buitron (2015). Inequality and labor market institutions (Staff Discussion Note, no. SDN/15/14). International Monetary Fund. https://www.imf.org/external/pubs/ft/sdn/2015/sdn1514.pdf.
  56. Jung, H. J., and M. Lease (2012). Improving quality of crowdsourced labels via probabilistic matrix factorization. In Proceedings of the 4th Human Computation Workshop (HCOMP) at AAAI, Toronto ON, Canada, 22–23 July 2012. Palo Alto: AAAI Publications, pp. 101–106. https://www.aaai.org/ocs/index.php/WS/AAAIW12/paper/viewPaper/5258.Google Scholar
  57. Kittur, A., J. V. Nickerson, M. Bernstein; et al. (2013). The Future of Crowd Work. In CSCW ‘13. Proceedings of the 2013 Conference on Computer Supported Cooperative Work, San Antonio, Texas, USA, 23–27 February 2013. New York: ACM, pp. 1301–1318.  https://doi.org/10.1145/2441776.2441923.Google Scholar
  58. Kochhar, S., S. Mazzocchi, and P. Paritosh (2010). The Anatomy of a Large-scale Human Computation Engine. In HCOMP ‘10. Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, USA, 25 July 2010. New York: ACM, pp. 10–17.  https://doi.org/10.1145/1837885.1837890.Google Scholar
  59. Kuek, S. C., C. Paradi-Guilford, T. Fayomi; et al. (2015). The Global Opportunity in Online Outsourcing (World Bank Other Operational Studies, no. 22284). The World Bank. https://econpapers.repec.org/paper/wbkwboper/22284.htm. Accessed 2 January 2018.
  60. LaPlante, R., and S. Silberman (2015). Design notes for a future crowd work market. 13 February 2015. https://medium.com/@silberman/design-notes-for-a-future-crowd-work-market-2d7557105805. Accessed 3 January 2018.
  61. LaPlante, R., and M. S. Silberman (2016). Building trust in crowd worker forums: Worker ownership, governance, and work outcomes. In Weaving Relations of Trust in Crowd Work: Transparency and Reputation Across Platforms, workshop at WebSci ‘16, Hannover, Germany, 22–25 May 2016.Google Scholar
  62. Lease, M., J. Hullman, J. Bigham, et al. (2013). Mechanical Turk is Not Anonymous (SSRN Scholarly Paper, no. ID 2228728). Rochester: Social Science Research Network. https://papers.ssrn.com/abstract=2228728. Accessed 3 January 2018.
  63. Lee, M. K., D. Kusbit, E. Metsky; et al. (2015). Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers. In CHI ‘15. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, April 18–23, 2015. New York: ACM, pp. 1603–1612.  https://doi.org/10.1145/2702123.2702548.Google Scholar
  64. Liebman, W. (2016). Es geht um Menschen, für die der Schutz des amerikanischen Arbeitsrechts nicht gilt (Six Silberman, Interviewer). youtube.com/watch?v=Dy8R3jAu8Fo. Accessed 13 May 2017.
  65. Mandl, I., M. Curtarelli, S. Riso, et al. (2015). New forms of employment (Research Report, no. EF1461). Eurofound.  https://doi.org/10.2806/989252.
  66. Mao, A., A. D. Procaccia, and Y. Chen (2013). Better Human Computation Through Principled Voting. In AAAI’13. Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, Bellevue, Washington, 14–18 July 2013. Palo Alto: AAAI Publications, pp. 1142–1148. http://dl.acm.org/citation.cfm?id=2891460.2891619. Accessed 2 January 2018.Google Scholar
  67. Marder, J., and M. Fritz (2015). The Internet’s hidden science factory. PBS NewsHour, 11 February 2015. https://www.pbs.org/newshour/science/inside-amazons-hidden-science-factory. Accessed 2 January 2018.
  68. Martin, D., B. V. Hanrahan, J. O’Neill; et al. (2014). Being a Turker. In CSCW ‘14. Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, Baltimore, Maryland, USA, 15–19 February 2014. New York: ACM, pp. 224–235.  https://doi.org/10.1145/2531602.2531663.Google Scholar
  69. Martin, D., J. O’Neill, N. Gupta; et al. (2016). Turking in a Global Labour Market. Computer Supported Cooperative Work, vol. 25, no. 1, pp. 39–77.  https://doi.org/10.1007/s10606-015-9241-6.CrossRefGoogle Scholar
  70. Mr Y Aslam, Mr J Farrar and Others -V- Uber. No. 2202551/2015 & Others (Employment Tribunal October 12, 2016). https://www.judiciary.gov.uk/wp-content/uploads/2016/10/aslam-and-farrar-v-uber-employment-judgment-20161028-2.pdf.
  71. O’Neill, J., and D. Martin (2013). Relationship-Based Business Process Crowdsourcing? In P. Kotzé et al. (eds): INTERACT 2013. Human-Computer Interaction – INTERACT 2013, Cape Town, South Africa, 2–6 September 2013. Berlin: Springer, pp. 429–446.  https://doi.org/10.1007/978-3-642-40498-6_33.Google Scholar
  72. Osborne, H. (2016). Uber loses right to classify UK drivers as self-employed. The Guardian, 28 October 2016. http://www.theguardian.com/technology/2016/oct/28/uber-uk-tribunal-self-employed-status. Accessed 1 January 2018.
  73. Ostrom, E. (2000). The future of democracy. Scandinavian Political Studies, vol. 23, no. 3, pp. 280–283.Google Scholar
  74. Pine, K. H, and M. Liboiron (2015). The Politics of Measurement and Action. In CHI ’15. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Seoul, Korea, 18 - 23 April 2015. New York: ACM, pp. 3147–3156.  https://doi.org/10.1145/2702123.2702298.
  75. Rani, U. (2017). Adapting social protection systems for workers in crowdwork platforms. Presented at the 5th Conference of the Regulating for Decent Work Network, Geneva, 3–5 July 2017. Geneva: International Labour Office.Google Scholar
  76. Rao, H., S.-W. Huang, and W.-T. Fu (2013). What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task. In First AAAI Conference on Human Computation and Crowdsourcing, Palm Springs, CA, USA, 7–9 November 2013. Palo Alto: AAAI Publications. https://www.aaai.org/ocs/index.php/HCOMP/HCOMP13/paper/view/7525. Accessed 2 January 2018.Google Scholar
  77. Raval, N., and P. Dourish (2016). Standing Out from the Crowd: Emotional Labor, Body Labor, and Temporal Labor in Ridesharing. In CSCW ‘16. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San Francisco, California, USA, 27 February - 02 March 2016. New York: ACM, pp. 97–107.  https://doi.org/10.1145/2818048.2820026.Google Scholar
  78. Retelny, D., S. Robaszkiewicz, A. To; et al. (2014). Expert Crowdsourcing with Flash Teams. In UIST ‘14. Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, Honolulu, Hawaii, USA, 05–08 October 2014. New York: ACM, pp. 75–85.  https://doi.org/10.1145/2642918.2647409.Google Scholar
  79. Sarasua, C., and M. Thimm (2014). Crowd Work CV: Recognition for Micro Work. In L. M. Aiello and D. McFarland (eds): SocInfo 2014.. Social Informatics, Barcelona, Spain, 11 November 2014. Springer, Cham, pp. 429–437.  https://doi.org/10.1007/978-3-319-15168-7_52.Google Scholar
  80. Schmidt, F. A. (2016). Arbeitsmärkte in der Plattformökonomie — Zur Funktionsweise und den Herausforderungen von Crowdwork und Gigwork (Project Report). Bonn: Friedrich-Ebert-Stiftung.Google Scholar
  81. Siddiqui, F. (2017). Uber triggers protest for collecting fares during taxi strike against refugee ban. Washington Post, 29 January 2017. https://www.washingtonpost.com/news/dr-gridlock/wp/2017/01/29/uber-triggers-protest-for-not-supporting-taxi-strike-against-refugee-ban/. Accessed 2 January 2018.
  82. Silberman, M. S.; and L. C. Irani (2016). Operating an Employer Reputation System: Lessons from Turkopticon, 2008-2015. Comparative Labor Law and Policy Journal, vol. 37, no. 3. https://cllpj.law.illinois.edu/archive/vol_37/.
  83. Silberman, M. S., L. Irani, and J. Ross (2010a). Ethics and Tactics of Professional Crowdwork. XRDS, vol. 17, no. 2, pp. 39–43.  https://doi.org/10.1145/1869086.1869100.CrossRefGoogle Scholar
  84. Silberman, M. S., J. Ross, L. Irani, et al. (2010b). Sellers’ Problems in Human Computation Markets. In HCOMP ‘10. Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, USA, 25 July 2010. New York: ACM, pp. 18–21.  https://doi.org/10.1145/1837885.1837891.
  85. Silberman, M. S., E. Harmon, L. Irani, et al. (2017). Crowd work and the “on-demand” economy. Hesamag, no. 16, pp. 36–39.Google Scholar
  86. Silberman, M. S., B. Tomlinson, R. LaPlante, et al. (2018). Responsible research with crowds: pay crowdworkers at least minimum wage. Communications of the ACM, vol. 61, no. 3, pp. 38–41.  https://doi.org/10.1145/3180492.CrossRefGoogle Scholar
  87. Smith, A. (2016). Gig Work, Online Selling and Home Sharing (Research Report). Pew Research Center. http://www.pewinternet.org/2016/11/17/labor-platforms-technology-enabled-gig-work/. Accessed 3 January 2018.
  88. Snow, R., B. O’Connor, D. Jurafsky, et al. (2008). Cheap and Fast—but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks. In EMNLP ‘08. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, pp. 254–263. http://dl.acm.org/citation.cfm?id=1613715.1613751. Accessed 2 January 2018.
  89. Söderqvist, F. (2016). Plattformsekonomin och den svenska partsmodellen (Report). Stockholm: Unionen.Google Scholar
  90. Sriraman, A. (2016). CrowdCamp Report: Protecting Humans – Worker-Owned Cooperative Models for Training AI Systems. Follow the Crowd, 29 November 2016. https://humancomputation.com/blog/?p=9502. Accessed 3 January 2018.
  91. Stewart, J. (2017). The Human Army Using Phones to Teach AI to Drive. WIRED, 9 July 2017. https://www.wired.com/story/mighty-ai-training-self-driving-cars/. Accessed 2 January 2018.
  92. Stone, K. V. W. (2006). Flexibilization, Globalization, and Privatization: Three Challenges to Labour Rights in Our Time. Osgoode Hall Law Journal, vol. 44, pp. 77.Google Scholar
  93. Taylor, Alex (2015). After Interaction. Interactions, vol. 22, no. 5, pp. 48–53. https://doi.org/10.1145/2809888.CrossRefGoogle Scholar
  94. Upwork (2016). User Agreement. 12 September 2016. https://www.upwork.com/legal/. Accessed 15 June 2017.
  95. Valentine, M. A., D. Retelny, A. To et al. (2017). Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In CHI ‘17. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, Colorado, USA, 06–11 May 2017. New York: ACM, pp. 3523–3537.  https://doi.org/10.1145/3025453.3025811.
  96. Vannette, D. (2017). Using Attention Checks in Your Surveys May Harm Data Quality. Qualtrics. 28 June 2017. https://www.qualtrics.com/blog/using-attention-checks-in-your-surveys-may-harm-data-quality/. Accessed 3 January 2018.
  97. Whiting, M. E., D. Gamage, S. N. S. Gaikwad, et al. (2017a). Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms. In CSCW ‘17. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, Oregon, USA, 25 February - 01 March 2017. New York: ACM, pp. 1902–1913.  https://doi.org/10.1145/2998181.2998234.
  98. Whiting, M. E., D. Gamage, S. Goyal, et al. (2017b). Designing A Constitution for a Self-Governing Crowdsourcing Marketplace. Presented at the Collective Intelligence Conference, Brooklyn, NY, USA, 15–16 June 2017.Google Scholar
  99. Wood, A., M. Graham, and V. Lehdonvirta (2016). The new frontier of outsourcing: online labour markets and the consequences for poverty in the Global South. Presented at the Work, Employment and Society Conference, Leeds, UK, 6–8 September 2016.Google Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Encountering TechLLC and Portland State UniversityPortlandUSA
  2. 2.IG Metall (German Metalworkers’ Union)Frankfurt am MainGermany

Personalised recommendations