Worker Perspectives on Designs for a Crowdwork Co-operative

Abstract

Crowdwork platforms such as Amazon Mechanical Turk (AMT) are a crucial infrastructural component of our global data assemblage. Through these platforms, low-paid crowdworkers perform the vital labour of manually labelling large-scale and complex datasets, labels that are needed to train machine learning and AI models (Tubaro et al., Big Data & Society, 7(1), 2020) and which enable the functioning of much digital technology, from niche applications to global platforms such as Google, Amazon and Facebook.

In this chapter, we reflect on how a ‘design justice’ approach might be valuable to build on insights gained from a series of exploratory discussions we have engaged in with US-based crowdworkers about how a crowdworker co-operative might work in practice, and begin to sketch out a potential software architecture that could form the basis of future participative approaches to the design and development of a crowdworker co-operative.

We begin by describing and reflecting on our own evolving methodology and how it fits with the ‘design justice’ lens we propose for future work. Following this, we present findings from our discussions with crowdworkers about how a crowdwork co-operative might work in practice, including what values workers would like to see embedded in the design. We then finish with the outline of a prototype software architecture for a crowdworker co-operative that could be used as a starting point in future design work in collaboration with crowdworkers.

Introduction

Crowdwork platforms such as Amazon Mechanical Turk (AMT) are a crucial infrastructural component of our global data assemblage. Through these platforms, low-paid crowdworkers perform the vital labour of manually labelling large-scale and complex datasets, labels that are needed to train machine learning and AI models (Tubaro et al., 2020) and which enable the functioning of much digital technology, from niche applications to global platforms such as Google, Amazon and Facebook.

While some social enterprise crowdwork platforms exist (Gray & Suri, 2019), the most popular platforms such as AMT are organised so that individual workers compete to be able to complete microtasks—or Human Intelligence Task (HITs)—advertised on the supposedly ‘neutral’ platform by requesters, most of whom pay far below minimum wage rates for the completion of tasks. The platform also takes a proportion of what the requester pays.

Crowdwork is often imagined as a solitary and isolating experience; however, researchers such as Gray et al. (2016) identify that crowdworkers often collaborate to meet their social and technical needs through, for example engaging in online forums. Beyond worker online forums, various socio-technical interventions in crowdwork infrastructures have been developed, including a number of popular browser plugins used by workers to help manage workflow. Most notable among these is Turkopticon, a browser plugin that enabled workers to review requesters, developed by Lili Irani and colleagues and actively maintained from 2008 to 2019.

In many ways, workers’ efforts to adapt the dominant crowdwork infrastructure can be understood as them engaging in an act of bricolage aimed at re-constituting infrastructure to produce the most efficient workflow possible from the tools available. Nevertheless, despite such activity, significant barriers stand in the way of crowdworkers’ efforts to collectively improve their labour conditions and Graham et al. (2017, p. 146) observe a “race to the bottom” in relation to worker pay and conditions.

Crowdworker co-operatives have been mentioned by a number of researchers and labour activists as a possible alternative, but have not yet been explored in detail (Gray & Suri, 2019; Graham et al., 2017; Scholz, 2016). In this chapter, we reflect on how a ‘design justice’ approach might be valuable to build on insights gained from a series of exploratory discussions we have engaged in with US-based crowdworkers about how a crowdworker co-operative might work in practice, and begin to sketch out a potential software architecture that could form the basis of future participative approaches to the design and development of a crowdworker co-operative.

We begin by discussing recent research on the possibility of ‘platform co-operatives’, including crowdwork co-operatives. We then go on to describe and reflect on our own evolving methodology and how it fits with the ‘design justice’ lens we propose for future work. Following this, we present findings from our discussions with crowdworkers about how a crowdwork co-operative might work in practice, including what values workers would like to see embedded in the design. We then finish with the outline of a prototype software architecture for a crowdworker co-operative that could be used as a starting point in future design work in collaboration with crowdworkers.

Platform Co-operativism

Different forms of co-operative, including worker and consumer co-operatives, have existed since the 1800s, as an alternative to strictly capitalist relations of labour and consumption. Many co-ops follow the Rochdale Seven Principles, originally formed by the Rochdale Society of Equitable Pioneers in the mid-1800s, and most recently updated in 1995. These are as follows:

• 1. Voluntary and Open Membership

• 2. Democratic member control

• 3. Member economic participation

• 4. Autonomy and independence

• 5. Education, training and information

• 6. Cooperation among co-operatives

• 7. Concern for community

More recently, in response to the advances of “platform capitalism” (Srnicek, 2017), people have begun to imagine how co-operative principles and practices might be adapted to address the forms of capitalist exploitation evident in the digital economy and which provide “an alternative to venture capital-funded and centralized platforms”. Scholz (2016) and the Platform Co-operative Consortium, of which he is a member, have begun work to conceptualise different possible models of platform co-operative across the wider platform economy that includes crowdwork.

The Platform Co-operative Consortium proposes a “platform co-operative” should be based on principles including:

Broad-based ownership of the platform, in which workers control the technological features, production processes, algorithms, data, and job structures of the online platform; Democratic governance, in which all stakeholders who own the platform collectively govern the platform; Co-design of the platform, in which all stakeholders are included in the design and creation of the platform ensuring that software grows out of their needs, capacities, and aspirations; An aspiration to open source development and open data, in which new platform co-ops can lay the algorithmic foundations for other co-ops. (https://platform.coop/)

Scholz (2016) proposes an initial typology for beginning to think through possible designs for platform co-operatives, identifying (1) Co-operatively Owned Online Labour Brokerages and Market Places, (2) City-Owned Platform Co-operatives, (3) Producer-owned Platforms, (4) Union-Backed Labour Platforms and (5) Co-operatives from Within.

A platform crowdworker co-operative offering decent work opportunities independent of existing platforms such as AMT—as in (1) or (4) above—has significant advantages, including the co-operative being able to use all income to pay workers and reinvest in the co-operative rather than channelling funds to Amazon. Nonetheless, challenges abound in relation to how a co-operative start up might compete with a platform such as AMT for tasks for workers to complete and hiring programmers to develop and maintain the platform. An alternative to setting up an entirely independent platform from scratch is the type Scholz refers to as “Platform from Within”, which he describes as a form of “hostile takeover” in which “worker cooperatives form[] inside the belly of the sharing economy”. The example Scholz uses is Uber drivers using Uber’s technical infrastructure to run their own enterprise. Such a model is also possible with an infrastructure such as AMT—a worker co-operative could essentially ‘plug in’ to the AMT infrastructure to siphon off HITs, but have its own separate distribution and governance structures. Such an approach could be temporary while the co-operative develops the necessary expertise to disconnect from the AMT feed. However, despite the advantages of such an approach, depending on the specific actions of the co-operative, such activity may be against AMT terms and conditions, something which could be off putting for workers that do not want to risk being banned from AMT. These issues point to how the material conditions of workers and existing crowdwork infrastructures generate significant ambivalences around how co-operative models might be leveraged in efforts to address labour exploitation as a form of data power within the global AI assemblage.

Moving Towards a Critical Design Epistemology

This chapter, written in mid-2020, reflects on a moment in the unfolding of an interdisciplinary collaboration between the authors. How we are positioned as researchers in the field of crowdwork and how we came to work together is important for understanding the trajectory of our work and the nature of this particular contribution. Taking inspiration from Irani and Silberman’s (2016) reflections on the “wider economic and cultural imaginaries of design as a social role”, we here reflect on these challenges in relation to our own work.

While we have all long had interests in labour relations and capitalist modes of production, it was Alessandro that first became actively involved in research on paid crowdwork infrastructures. As a Computer Scientist (CS) working as a postdoctoral researcher on an EU funded project, Alessandro initially focused on experimental work with the goal of improving the efficiency and efficacy of crowdworker tasks, such as image recognition, labelling and annotation, text summarisation, translation, data cleaning and information retrieval. Unhappy with the surveillance-oriented methods of quality control, Alessandro used his CS expertise to develop a Gold Question (GQ) detector that if implemented could be used by crowdworkers to alert them to the existence of quality check questions within a task, and which with enough workers on board could potentially be used to initiate a digital strike. It was at this point that Alessandro invited Jo (a researcher in the field of Critical Data Studies [CDS]) to collaborate—in the first instance to help think through social theoretical lenses that could be used to frame this work on Gold Question detection.

After completing this initial work (Checco et al., 2018), we reflected on the lack of crowdworker voices in our nascent interdisciplinary collaboration and managed to secure a small amount of internal funding to pay a Research Assistant—Elli—to conduct a series of interviews with crowdworkers. These interviews explored a range of issues experienced by crowdworkers, as well as exploring their perceptions on the GQ detector and what ideas they had for improving crowdworker conditions. We decided to engage with US-based crowdworkers as we were particularly interested in how the increasingly global nature of the crowdworker labour market was being experienced by workers in countries with higher costs of living.

Emerging from some of our early interviews was the idea of a co-operative model for crowdwork. This idea of a crowdwork co-operative was also something that Alessandro had begun to explore from a Computer Science perspective, and we decided to begin actively asking crowdworkers about the co-operative idea in the second stage of interviews. Despite gathering significant insights from crowdworkers about the potential of different socio-technical interventions in the crowdwork infrastructure, our reflections on our evolving interdisciplinary approach raised questions about how the crowdworkers were included in the design of potential socio-technical interventions such as the design of a crowdwork co-operative.

Our research team discussions surfaced concerns about how we had begun by imagining the crowdworkers we spoke to as temporary participants in our research, rather than as co-designers of alternative digital labour infrastructures. While we had engaged with some crowdworker forum moderators and activists alongside conducting the interviews, the direction of the project remained relatively researcher-led.

This was in part due to how our own understanding of what we were trying to achieve unfolded over the years, but also the material constraints on our work. For example, we only had a small amount of internal funding which meant we could only compensate 20 crowdworkers for an hour of their time—an important ethical consideration when relying on the time and energy of very low-paid workers. Funding limitations also meant we were constrained in our ability to travel to meet crowdworkers and the number of languages we had at our disposal to interview crowdworkers. Other commitments in our work and lives also meant constraints in terms of the number of hours we as researchers could dedicate to engaging with the crowdworkers and on the project. Nonetheless, while we were cognisant of the need to recognise how the material realities of both the crowdworkers and researchers constrain practice, we still wanted to avoid our engagement with crowdworkers being ‘tokenistic’ (Arnstein, 1969) and instead try and foster a practice in any future work in which the locus of control would be with the crowdworkers.

Through our reflections on these issues, we began to explore ideas around critical design approaches. Critical design researchers have long considered how critical interventions within design might be undertaken. As Bardzell et al. observe, a key decision in any critical design project is understanding what it is about the world you are trying to “provoke” (2012, p. 290). For Bardzell et al. (2012) the answer was the gendering of space; for our research team it was what we perceived to be an exploitative capital labour relation at the core of the crowdwork infrastructure. While Bardzell et al.’s (2012) critical design process involved gathering reaction to ‘provocative designs’ that they embedded in gendered spaces (e.g. the home and locker room), our approach had been to garner the reaction of crowdworkers to ideas for ‘provocative designs’ (the Gold Question detector and crowdwork co-operative) that would disrupt the existing crowdwork infrastructure and the form of capitalist labour relation at its core.

In total, we spoke to 20 US-based crowdworkers over a period of 18 months. These interviews were conducted from the UK by Skype. We recruited participants via crowdworker forums and Slack channels, including seven who had previously taken part in experimental work conducted by Alex and a colleague in Computer Science on crowdwork quality control (three workers) and crowdworker cooperation (four workers). Of these, the idea for the crowdworker co-operative emerged in discussions with four out of the first ten interviews. We then actively asked about the crowdwork co-operative intervention in the final ten interviews.

Through this process of engaging with crowdworkers, we came to understand more about how US-based crowdworkers understood the crowdwork labour relation and their preference for constructive design interventions such as the ‘worker co-operative’ idea, rather than design ideas that some perceived as confrontational such as the ‘Gold Question Detector’ (Checco et al., 2018). However, what to do with this insight remained somewhat unclear until we considered emerging ideas relating to ‘design justice’ (Costanza-Chock, 2020).

Building on work in the fields of Participatory Action Research and co-design, and the ideas and practices of activists in the US-based Design Justice Network, Costanza-Chock argues the case for a design justice framework that extends beyond common framings of interventionist design paradigms such as ‘social impact’ and ‘design for good’. Recognising “communities to be co-researchers and co-designers, rather than solely research subjects or test users”, Costanza-Check presents a design justice framework that addresses questions of (1) values encoded in the design of systems and objects, (2) practices relating to who is engaged in and controls design processes, (3) narrative about how things are designed and how design problems are scoped and framed, (4) sites at which design takes place and how accessible they are by those most impacted by design processes and (5) pedagogies relating to teaching and learning about design justice (2020, pp. 35–36).

In writing this chapter, we were inspired by this framework in a number of ways:

• The writing style we decided to adopt was motivated by points (3) and (5) about narrative and pedagogies. We decided to produce an honest and reflective account of our own practices as an emergent interdisciplinary team of researchers and our intervention within the crowdwork space, both as an effort to reflect on our own narrative and also as a pedagogical intervention in terms of developing our own learning about our practice while also producing a written artefact that could be used in other contexts of learning and teaching.

• The empirical work we present in the following section is guided primarily by (1). While our discussions with crowdworkers had confirmed our understanding of existing crowdwork infrastructures as embedding an exploitative capital-labour relation albeit experienced in different ways by different crowdworkers, we also took from these discussions a more detailed understanding of the values that US-based crowdworkers perceived as important to the design of an alternative and fairer crowdwork infrastructure.

We then began to think through how we might practically adopt these values to sketch out an initial prototype of what a crowdwork co-operative infrastructure might look like and which might be used as a starting point in future co-design activity. We began by incorporating lessons from Computer Supported Cooperative Work (CSCW), an approach that builds on 1980s Participatory Design, which stems from the Scandinavian tradition of trade union collaboration, and the second wave of Activity Theory. CSCW is an interdisciplinary field aimed at studying computer-assisted activities that involve people coordination; it therefore seemed appropriate for beginning to think through some practical concerns that may need to be considered in the ongoing co-design of a dynamic crowdwork co-operative infrastructure.

Work in the field of CSCW and cognate fields has recognised a number of important features of technology-assisted co-operative work that could be pertinent to consider in efforts to co-design a dynamic crowdwork co-operative prototype. These include the following:

• The recognition of the existence of multiple and conflicting incentives and disincentives in coordinated work, at the institutional, organisational and group level, and affecting differently the hierarchies embedded within each of these levels (e.g. competition might be more present amongst junior members, while collegial behaviour is more accepted at senior level) (Pratt et al., 2004).

• The recognition that co-construction is usually delegated to a higher class of workers, while lower-class workers are delegated to routine work, with a minimal input in the decision process (Bardram, 1998a). It is thus necessary to provide digital spaces to allow a democratic co-construction phase that typically cannot happen during the routine work setting.

• The tension between work routine and the need for change for the sake of efficiency (Pratt et al., 2004).

• The cooperation breakdowns and exceptions are the salient points in which negotiation and the establishing of new work heuristics occurs—temporary failures of cooperation should be acknowledged as part of the learning and co-construction process (Bardram, 1998a; Bødker et al., 1988).

• Recognition of the importance of interrogating the dualism between competition and cooperation (Malone & Crowston, 1990).

• The importance of awareness (mutual or one-directional) of one individual’s activity by other members, enhancing this awareness when needed is what makes collaborative systems successful (Carstensen & Schmidt, 1999; Pratt et al., 2004).

• The recognition that the design process of a manufacture-like process is relatively easy; however, the creative part of the interaction and the handling of interdependencies are characterised by an “overwhelming complexity” (Carstensen & Schmidt, 1999; Bødker et al., 1988): in our prototype we found this reflects difficulties in modelling recruitment, work division and meta-cognitive interactions.

In conclusion, emerging from critical design, CSCW and cognate fields is the clear understanding that we need to be aware of the risk of designers relegating workers to a non-creative routine, after only an initial phase when co-construction was informed by them (Bardram, 1998b). Similarly, design has a political effect that risks presenting the designers as “saviors descending to help others”, shifting the focus away from the unjust system of value distribution of the platform economy (Irani & Silberman, 2016).

Inspired by our reading of critical design, design justice and CSCW approaches, the software architecture ‘artefact’ that we produced (Section “Crowdworker Perspectives on Crowdwork Co-operatives”) is purposefully partial and contestable, and aims to embed the above insights from CSCW about the design of technology-assisted activities that involve the coordination of people, as well as the perspectives of the crowdworkers we spoke to as presented in the following section. We envisage the prototype as a possible starting point for future critical design practices that engage crowdworkers in different ways and across sites, in ways accessible in relation to material constraints such as time, resources and mobility.

Crowdworker Perspectives on Crowdwork Co-operatives

MTurk somehow seems to promote the idea that ten cents a minute is standard pay. And I don’t know who tried to live on ten cents a minute. (I7)

The crowdworkers we engaged with reported a wide range of challenges in their work, many of which reflected the findings of previous research (e.g. Hara et al., 2018; Martin et al., 2014; Ross et al., 2010; Kittur et al., 2013; Fort et al., 2011). In particular, they talked about the increasing number of workers on the platform resulting in reduced work availability and declining pay, their fear of having work rejected by the requester and therefore not receiving payment for work completed, unpaid time spent looking for good work on the platform, poor communication with requesters, and workers’ lack of power to resolve issues and improve conditions.

All the crowdworkers that we spoke to directly about the crowdwork co-operative idea—both those that were positive about it and those that had some reservations—were enthusiastic and curious to explore what working in a crowdwork co-operative platform might be like.

Two participants (I13, I15) were very enthusiastic:

Yes, that would, right there, solve so many problems, so many problems … the humanitarian in me, I’d be like, “Yeah, all over that”. (I15)

Some perceived advantages around the possible culture shift that working in cooperation with other workers might engender within the currently hyper-competitive crowdwork culture:

[I]t’s really competitive and that puts people into a state of, you know, I have to protect myself over the next person. So when you take that threat away, when you give support where there was never any in certain areas, you’re going to see a shift within that paradigm. (I15)

On the other hand, other workers did have reservations that would need to be addressed in the co-operative design. They perceived that the most pressing concerns that any crowdwork co-operative design would need to address were work availability, quality of work completed, dealing with issues of free riders and workers without appropriate skills, for example English language, and convincing the most successful workers and good requesters to join the co-operative.

Despite these reservations, as will be explored in the following sections, workers had many ideas for how challenges might be addressed and said that if they could be sure the work and payment distribution was fair and if there was enough work available to meet their income target they would be interested in trying to be part of a crowdwork co-operative.

The next section lays out the key issues that crowdworkers perceived would be fundamental to the design of a successful crowdworker co-operative.

Towards a Worker-Driven Design for a Crowdwork Co-operative

Co-operative Values from the Workers’ Perspective

Across the different crowdworkers we spoke to, we identified a broad set of values that they imagined might be embedded within the design of a co-operative infrastructure for crowdwork. These included the following:

• 1. Fairness in payments—with their views reflecting notions of distributive fairness and procedural fairness as discussed by Fieseler et al. (2019)

• 2. Democratic or collective decision making, with an emphasis on equal representation (one person one vote)

• 3. Horizontal governance structure—they often recognised some coordination was required, but believed any ‘manager’ role should be equal to other workers in terms of power and reward

• 4. Transparent communication within the co-op

• 5. Open information and communication between workers and requesters

• 6. Accountability for both requesters and workers, including accountability for quality of work produced

• 7. Camaraderie and a sense of community, trust, mutual assistance and cooperation

• 8. Platform design based on workers’ needs

• 9. Empowerment of workers in the co-operative to decide membership and rules

• 10. Security of work

• 11. Commitment to the co-operative

The broad set of values sit comfortably with the Rochdale principles (International Cooperative Alliance, 1995) mentioned in “Platform Co-operativism” section. While some of the values are more specific to the crowdwork context, for example, platform design based on workers’ needs, the more general values of fairness in payment, democratic decision making, horizontal governance and empowerment of workers align with the Rochdale principles. The only Rochdale principles that are missing from what workers explored was ‘co-operation among co-operatives’ and the education and training aspects of ‘education, training and information’, although workers did discuss mutual support among workers for self-directed learning. They were not directly asked about these two issues, and it is something that could be explored further in the future.

Underlying this broad set of values, workers identified a number of specific ideas about what they imagined would be important practical components of a crowdwork co-operative design. These can be broken down into four thematic areas: (1) platform infrastructure, (2) payment, (3) quality control and (4) decision making and governance. Of these the first—platform infrastructures—is most specific to the material conditions of crowdwork, whereas the rest reflect the types of practical concerns that would likely be seen in any type of co-operative. Each of these themes was embedded within an overarching meta-theme of empowering workers. Table 1 identifies the different ideas generated by crowdworkers in relation to each of these themes. All ideas are detailed in the subsequent section.

Table 1 Key ideas for the design of a crowdwork co-operative from a worker perspective

Platform Infrastructure

Participants had a number of suggestions about how to optimise the platform infrastructure of the co-operative to help with the distribution and efficient completion of work, to empower workers and to help make working on the platform feel more personal and engaging.

A key suggestion made by four workers (I9, I10, I11, I13) was to have worker profiles integrated into the platform. This is in stark contrast to the existing AMT platform in which workers are anonymous and are only represented by a worker identity number. They thought that profile information such as basic demographics, level of education, skills and work history (types and number of HITs completed) could be useful for a number of reasons. Two workers thought that having such a profile which listed their skill set would make the platform feel more humanised for them, rather than them just being a string of numbers.

[S]omething where I’m not just—,[…], where I’m not just that worker ID number. (I13)

A number of workers also thought worker profiles would help the co-operative divide workers into subgroups based on their skill sets, help with filtering work in order to allocate it to the right workers and help workers quickly have a good sense of their role in the co-op [I9, I10, I13, I18]. Filtering work this way would make finding and completing tasks more time and cost effective and help avoid taking the same tasks twice, which is a significant inefficiency of the current infrastructure where all workers spend a lot of time searching for HITs independently. Two workers also thought worker profiles could help make sure malicious workers were not able to undertake tasks not appropriate for them [I9, I12].

As well as worker profiles, four participants [I5, I10, I11, I12] also recommended that requesters have a profile on the platform, both to make the process more personal and so workers would be able to find requesters in order to follow them if they like the work they provide and in order to check their duration on the platform. As well as requester profile, requester ratings were also suggested. While existing platforms such as AMT do not have a requester rating functionality many workers do use browser plugins such as Turkopticon to check requester ratings (Irani & Silberman, 2013); however, this plugin is no longer being supported by the developers. The most recommended [I10, I12, I16, I20] idea was to establish a rating system built directly into the platform similar to other apps, like Lyft and Uber, where the workers could rate the requesters and leave reviews based on the amount they pay and their task practices. Workers also recommended having a filter in the platform that would block requesters if they fall below a certain rating level or if they offer payment below a commonly agreed threshold [I4, I5].

[I]t would be cool if on a site they could be basically removed at that rate [of pay] … for example, there should be like an override that says that if more than ten workers say that then we—Block the requester. (I4)

Beyond profiles, the workers [I4, I5, I12, I20] we spoke to suggested that the co-operative platform could directly integrate the various different scripts and tools that workers currently use as, for example browser plugins. This would help workers in the co-operative to find good work and work more efficiently from directly within the platform.

[i]f you took all these features like you have Turkopticon and Hit Forker, and everything else designed to help workers find good work and help you out, you know if that was incorporated into the site to begin with. (I4)

They also suggested that a worker communication channel be embedded within the platform [I5, I9, I10, I15, I19, I20]. Currently workers use a variety of different forums and Slack channels to communicate with one another; however, they perceived that this communication would be better if integrated directly into the platform, creating a central hub of information and allowing requesters to participate in conversations.

I would like it to have a more solidified community rather than the scattered forums. (I9)

I10 and I15 also discussed how these integrated communication channels could be extended to communications with requesters—for example, by including a chatbox next to tasks through which workers could report on any ambiguities. They perceived this would help decrease rejection rates and help requesters weed out malicious workers instead of rejecting workers in bulk.

Payment

Currently crowdworkers get paid a piece rate for work completed, with no payment for time spent searching for work or on tasks rejected by the requesters. Seven of the workers we spoke to thought the co-operative should shift to a payment model in which the amount paid reflects time spent working rather than tasks completed [I5, I12, I13, I16, I17, I18, I20], and that this would be a fairer reflection of the work undertaken to find and complete different types of HITs. They recommended having a system in place to log each worker’s contribution. I12 suggested the payment to be calculated per minute in order to keep things as simple as possible.

That would have to be logged somehow. So you would have to be able to either have it logged by the platform or have people check in and check out when they are working. But then at the end of any given project, let’s say you’d have 20 workers who all had their varying amounts of time logged in, and then you would take whatever the fee was for that and divide it based on the time. (I17)

However, some workers also questioned if such standards for pay would be possible if work was scarce:

[H]ow would you guarantee, you know, like $8 an hour if you have a bad day where there’s no work for anybody, you know? (I19)

This issue of work scarcity on the ability of the co-operative to pay members was a concern for other workers. Some argued that the co-operative might struggle to attract requesters away from platforms such as AMT, particular if the co-operative was perceived by requesters as a form of unionising [I20]. Others were concerned about how any scarcity issues might impact on the culture of cooperation a co-operative would be dependent upon:

[T]hat that kind of scarcity makes people hungry. And when you’re hungry you’re not very cooperative, [laughs]. (I17)

Some workers also discussed how workers might be graded and whether requesters could pay a premium for high-quality, experienced workers [I5, I9, I10, I11, I13]. Some criticised the current ‘Masters’ system on AMT as lacking in transparency and argued that a transparent training and rating system in which workers could progress through different “quality tiers” [I15] would be beneficial to managing the co-operative and motivating workers.

There was no consensus among workers about whether higher quality or faster workers should receive more payment. However, a number of participants [I19, I11, I16, I17] recognised that the co-operative might not attract high-quality workers if they would have to share their income with the rest of the group and that it would therefore be challenging to have a standard hourly wage if the pace of work was not equal among members.

How to foster efficiency and trust within the co-operative was therefore another concern. Workers [I10, I12, I14, I16, I17] were concerned that some people might take a longer time to complete tasks perhaps because they were slower and less efficient at completing their work, or because they purposefully wanted to spread out the same amount of work over a longer period of time in order to increase the number of hours they received pay for.

Any system you put in place, eventually somebody’s going to try to find a way to abuse it, so you have to kind of find a way to safeguard before that happens, I guess. (I16)

Quality Control

One way to encourage both workers and requesters to an independent co-operative platform would be if the workers were trusted by requesters, and one another, to efficiently complete high-quality work. Workers recognised that some form of quality control process would be required by the co-operative. They suggested ideas for possible peer rating and review systems to try and identify poor quality work and people trying to abuse the system.

For example, I18 suggests to have a peer rating system among the workers where they would indicate if the other members were pulling their weight. The system would aggregate the reviews, and if the worker fell below a threshold they could be removed from the platform.

I think if people had an anonymous way to, you know, rate everybody on the team, and if you had, you know, certain kind of thresholds, where somebody was doing not a good job. (I18)

I17 recommended having some form of statistic of how many units of the same project have other workers completed in a specific amount of time so as to identify if someone is misusing the system. Others discussed how starting small as a co-operative might also help with this issue. I15, for example, thought starting with a small group of workers and building it slowly would make it easier to find malicious workers, and others observed that starting small would also help to create a sense of community and help address governance issues in which a consensus was required [I15, I19, I20].

Beyond how to manage work quality in a way that would foster fair payment and encourage requesters to use the platform, we also explored whether a co-operative might be able to offer work benefits that are not currently available to crowdworkers. While some workers perceived themselves as self-employed and therefore responsible for their own pension and health- and sickness-related benefits, some did believe that it would be good to receive such benefits through their work on the platform. However, they thought given the downward trend in how much requesters are willing to pay for HITs this was unlikely.

It’s great to have the option. … But, a lot of requesters have realised that there will always be people doing every hit regardless of how bad it’s paid, regardless of anything. … So, not only do I not expect for things like benefits and pension to come along, I actually expect this to be worse and worse paid compared to a normal job. (I20)

Given the co-operative would be competing against other platforms for requesters, it is likely that if the co-operative was to be able to pay a decent wage with benefits then regulatory action may also be required to enforce minimum standards. However, this is not something all crowdworkers are supportive of.

Decision Making and Governance

Workers thought that decision making in the co-operative should be done collaboratively, with each member of the co-operative having an equal vote [I10, I13, I16, I20]. However, one person did question what might happen if ‘free riders’ outnumbered productive workers in this model and thought something would need to be put in place to avoid that scenario:

[S]ay there’s five workers, John’s one doing most of the work, but the other four would like to be, you know, paid the same amount as John, they—, again, they could outnumber him with a vote [laughs] and, you know, get paid the same. So you would … just have to find a way to make it fair, where people can’t abuse it. (I16)

The question of how the co-operative might come to a consensus on issues was raised as potential concern by some, particularly if the co-operative grew beyond a small ground of workers:

If it was a smaller [number of people] it would be easier to come to a consensus. Part of the problem with crowdwork is that you have so many opinions in the crowd that it’s really hard to come to a consensus. (I19)

In relation to this issue, some workers thought some sort of responsibility structure would need to be put in place to help manage the co-operative:

[I]f it’s, let’s say, 30–40 people, maybe it could be like a perfect democracy, and just everybody votes, but anything over that I would say, definitely there would need to be one or two people in charge of certain things. (I20)

Some sort of coordinator role was suggested by a number of workers [I15, I18, I19, I20]. However, they understood this role to be a facilitator role that would be equal with other workers in terms of power and pay, rather than being part of hierarchical management structure.

[S]omebody like organising the whole thing is great, and making sure everybody’s on the same page and stuff like that, but I wouldn’t think that it would need to be a position by itself. (I18)

This role was imagined as involving helping to organise the work in the co-operative, ensuring work was completed, checking on the workers’ progress, opening discussions about the rules in the co-op, promoting communication, helping resolve issues and being the voice of the co-operative.

One worker suggested that there could be more elected positions beyond a single coordinator:

[A]s the group grows there could be like some elected positions maybe, like maybe a supervisor, maybe like a treasurer or something like that. (I20)

Key issues that would need to be decided through such decision-making processes include issues of membership of the co-operative and how to fairly allocate work.

The membership of the co-operative was recognised as a crucial aspect of decision making by a number of participants. Four participants [I9, I10, I13, I15] suggested the members of the co-operative would need a way to control membership. I9, for example, suggested a selective process to decide who could become a member of the platform, and I10 recognised the platform would need a system in place to deal with malicious workers. Requirements of membership suggested by the workers included things such as only taking workers above a particular approval rating or level of experience on existing platforms [I13, I15, I20], language proficiency tests to help with the allocation of tasks that require a particular language [I10, I11]. I10 also recommended that workers pay a fee for people to join the co-operative platform. I20 perceived that having some sort of application process to join the co-operative would help new workers demonstrate their commitment to the platform and create a stronger sense of community. Two workers [I4, I9] also suggested that requesters should have to apply to join the platform.

Another issue one worker [I9] raised related to decision making in task distribution related to economic geography. This worker argued that tasks should be allocated on the basis of geography, with requesters having to use workers from the country that they were located and at an appropriate rate for the living costs of that country:

I think for a start having platforms that are dedicated to Europe and the US, where they get their workers. I mean part of me feels like that’s not fair or just, but I feel like if a requester is in the US or Europe they should probably be getting European or US workers. It feels like the outsourcing is a little unfair. (I9)

This kind of issue would be the kind of question the co-operative would need to grapple with through the kind of democratic decision-making processes workers described earlier.

A Prototype Software Architecture for a Crowdwork Co-operative

As identified by the workers above, the first phase of a crowdwork co-op would face the problem acquiring a critical mass of requesters and workers to be able to be meaningful as an independent platform. For this reason, we will first focus on the intermediate “platform from within” (Scholz, 2016) idea of using an existing external platform (e.g. AMT) to acquire jobs, augmenting the worker experience with collaboration and cooperation tools, although with recognition that in some cases this may be against platform Terms and Conditions. Further, this approach also raises the challenge that the ‘platform from within’ can be disrupted by AMT (or other) platform architectural changes and therefore depends on external maintenance. This poses a challenge as the researchers and developers who could help maintain the platform, as they often cannot guarantee continued input to ensure sustainability, particularly if they do not have ongoing funding or support for the work from their institution or an external funder—an issue that Kristy Milland has drawn attention to in workshop discussions, for example Aroyo et al. (2018).

Building on the ideas of crowdworkers described in the above sections, we use Stanoevska-Slabeva’s (2002) community-orientated design framework to identify the different components of a prototype software architecture that could be used as a starting point for future co-design practices with crowdworkers. This design framework is based on Schmid’s Media Reference Model (1999), which identifies four ‘views’ that are critical to understanding a software architecture: community, implementation, transaction and infrastructure. Here we focus primarily on the community view as this will be a core aspect of any co-design process; we also highlight some of the key aspects of the implementation and transaction views.

Community View. This view captures the identity-shaping features and other static elements of the organisational structure.

• Roles: These are roles that a member of the co-op (together with automated tools) can undertake.

  • Worker: This is the basic role, in which a member is completing a HIT.

  • Platform negotiator: A member, with the aid of automatic tools, interacts with the platform before or after a HIT is accepted. This role is responsible for adding information in the worker role view (see Fig. 2) and in the shared knowledge database (as explained later in the knowledge service). Members could be assigned these roles through election by the membership. This role could be divided into three sub-roles:

    • Job seeking (human + tools): This is the role of scouting for suitable HITs in the platform. This is a task that is usually repeated by all workers, and having a dedicated role would save significant time. An experienced member would be able to locate promising jobs or jobs that are a good match to the members’ skills. This role can also be assisted by automatic filters, with parameters decided by the members (e.g. payment threshold) and would be an improvement on the existing job-matching scripts, as suggested by the workers we spoke to. This role is important also to implement (even in a semi-automatic way) the boycott of requesters and batches.

    • Rule clarification: This is the role of contacting the requesters to clarify the rules of a batch. Currently, each worker will typically have to ask for some clarification from the requester before starting a batch, so having a dedicated role would save significant time. It would also increase the overall quality of the work, because all questions asked by the member assuming this role could be automatically propagated to all members working on the batch. Much of this role could be completely automated by a tool of rule clarification, where previously asked questions could be shared among members, but an experienced member could still have this role to catch important questions to ask when the job rules are ambiguous.

    • Rejection appeal (human + tools): This is a fundamental role, as a rejection of completed work can have a detrimental impact on workers. By monitoring the rejection rate and the workers’ feedback, a member with this role can quickly identify unfair rejections, promptly contact the requesters to demand a reversal and flag the batch as unreliable to prevent additional HITs being completed by the co-op.

  • Training: The training role could have two different ways of being implemented.

    • Indirect training: This is a role that any worker can have, even during the completion of a HIT: the co-op software could enable indirect messaging related to a batch while working so that each co-op member could leave notes on a batch for the rest of the members that encounter the batch.

    • Direct training: A worker could signal problematic tasks, and an experienced member could monitor these signals (together with workers’ performance) and intervene by providing support via chat and screen sharing.

  • Coordinator: A coordinator could interact with platform negotiators and workers, to dynamically select groups with similar skills/requirements, to create a critical mass of members working on the same batches. This would allow the members to increase the quality (training), efficiency (less job seeking) and the bargaining power of the co-op (rule clarification and rejection control).

    • Membership: A coordinator could also enforce the co-op rules and take care of new membership and revocation.

  • Metarole: This is a role necessary to decide co-op rules and policies, and change the software itself. All members could vote to assign roles and change the structure of the co-op, for example changing the co-op rules (see Table 1, column 2). This can be achieved by a collaborative decision-making mechanism that can include coordinator roles. Important examples of rules that require an agreement are membership rules, job allocation rules, worker/requester exclusion rules and payment distribution rules.

• Valid rules: A clear set of rules and their enforcement are necessary to establish trust (Preece, 2000). It would be necessary that the co-op members agree on a set of policies for the governance of the co-op, together with the parameters of the software itself. These rules could change over time with a voting process. A rule with a high impact on the shape of the co-op is the payment distribution scheme, which as discussed above would require deliberation and agreements among co-op members given the different ideas suggested by the workers we spoke to (see Table 1).

• Members’ identity: Members could have a profile that would help them present an identity to the community, as suggested by some of the workers we spoke to. This profile could include statistics, skills and preferences, and would help a member assuming the coordinator role to facilitate the job matching and training.

• Common language: Members use a common language and slang, originally inherited from the existing forums, and share a common vocabulary of terms related to the target platform, for example AMT.

Implementation View: This view contains the community processes that are the set of activities that can be performed by the co-op. We cannot list all processes here, but some of them are the membership process, the discussion process, the job flagging and training annotation process and so on.

Transaction View: This view describes the coordination and communication processes available in the co-op. They can be divided into the following:

• Knowledge services: To manage and use knowledge in the co-op. Some of this knowledge is obtained from workers signals (e.g. flag a job) and others automatically obtained from the platform (job search) and from the workers (aggregate job difficulty, worker performance). It is necessary to maintain a database for this.

• Intention services: To signal a member’s intention or need, like the request for training, the need to abandon a batch and so on.

• Negotiation services: To make decisions about membership, new policies, jobs to target, how to allocate work and so on. A notable concept that needs to be negotiated is the potential pay redistribution: this would allow workers to share and thus mitigate the risk of having an underpaid batch affecting one worker’s hourly wage. Similarly, solidarity tools like paid leave could be implemented.

We summarise this model in Fig. 1.

Fig. 1

Co-op software architecture model

Example of the Worker Seeking Role and Worker Role Views

In Fig. 2, we summarise the view of the main views a worker would use. These would need to be implemented as a browser plugin. The worker seeking view will visualise a (re)ranked list of batches based on the job allocation rules decided by the co-op: this ranked list is potentially different for each worker. In this view the workers will be able to visualise requester statistics and other information on the batch (e.g. required skills) obtained by workers that already selected this job, allowing an informed decision.

The worker role view will allow the worker to access, in addition to the external platform view (in red), to rule clarifications obtained by the corresponding role. Similarly, notes from other workers that already completed this batch will be displayed there. Moreover, the worker can communicate with other co-op members directly (ask for help) to obtain direct training. Finally, the worker can notify the co-op of potential problems with the job by flagging it.

Fig. 2

Worker seeking role and worker role views

Conclusion: Towards a Crowdwork Co-operative Prototype?

The above exploration of worker perspectives on a co-operative model for crowdwork platform design, and the resulting ideas for a prototype software architecture, aim to be a partial and contestable early step in exploring how workers and researchers might work together to re-imagine the organisation of the ‘crowd’ labour that contemporary AI systems are dependent upon.

The contribution draws upon insights from critical design and digital labour studies, to bring into focus the relevance of crowdwork platforms to Critical Data Studies. Emphasis in the Critical Data Studies field has tended to be on the expansion of datafication and outcomes for data subjects. Here, we draw attention to those people that labour within the infrastructures that support datafication processes, illuminate structures of labour exploitation that many contemporary AI systems are dependent upon and ask—with workers—how might these labour conditions be improved. In doing so, we also highlight the value of critical design studies and of interdisciplinary collaboration between the social and computer sciences, particularly as the focus of CDS expands from identifying instances of domination and exploitation resulting from deepening datafication, towards addressing the question of what can be done about it.

Through drawing on insights from different disciplinary perspectives as well as from the workers themselves, the ambivalences of data power and how to address it come into clearer focus. The material realities of workers’ economic needs combined with the constraints baked into the existing capitalist crowdwork infrastructure, as well as the limitations on researchers’ ability to guarantee sustainability of contributions within existing institutional arrangements, all interact to reinforce the complexity of the challenge. It is too early to understand how the recent COVID-19 pandemic will impact these issues. Certainly, the potential for sustainable contributions from university-based researchers in some countries will be further impacted by shifting priorities and reduced budgets. It is likely that the sharp increase in unemployment resulting from the pandemic may mean more people seeking work through crowdwork platforms (Moss et al., 2020), which could further push down wages if there is an increased supply of labour. Yet, on the other hand, it is also possible that with an increase in remote working more generally, researchers may make more use of platforms such as AMT to source research participants, thus increasing the demand on the platforms which may counter some of this effect. Any future research should therefore remain mindful of possible impacts of the pandemic on workers, requesters and researchers.

While our focus has been on co-operatives as a new way to organise ‘crowdwork’—whether independent of or ‘from within’ existing infrastructures—we conclude by adding that we do not envisage a crowdworker co-operative as a standalone solution to worker exploitation in crowdwork markets. Clearly, in a ‘platform from within’ model the co-operative would still be tightly—and potentially vulnerably—embedded within the capitalist data economy, and an independent platform would likely not have the economies of scale to compete successfully with AMT. Neither of these models addresses the systemic low-pay issues in crowdwork that would make it difficult for a co-operative to pay a living—or even minimum—wage. Also, we recognise that much of the work being undertaken by crowdworkers, such as labelling of machine learning datasets, contributes to a complex set of challenges around the adoption of machine learning and AI across various sectors. It is important that any future work on crowdwork co-operatives remains mindful of this context. Nonetheless, we perceive that a co-operative model for crowdwork could be a progressive intervention in the context of broader developments involving labour market regulation in the interests of workers, and an AI sector regulated in line with egalitarian and democratic values.