Crowdsourcing and Massively Collaborative Science: A Systematic Literature Review and Mapping Study

Increasing amounts of scientific data are being produced at an exponential rate, fueled by the development of high-throughput technology and the rapid growth of research capacity [1]. Thus, data discovery and reuse can be extremely difficult for a researcher working alone. Such processes have been highly individualized, labor-intensive, and error-prone and are not well supported by existing systems since automated reasoning approaches do not encompass the cognitive abilities of a human brain for tasks such as characterizing a field or discipline [2]. To address these challenges, scientists have leveraged the power of crowds and large communities of volunteers to perform tasks that no known efficient algorithms can yet solve. Crowdsourcing has been established as a computing paradigm intended to bridge the gap between machine and human computation [3]. When applied to tackle scientific problems, crowdsourcing can be characterized by openness to a large pool of researchers and citizen scientists and their respective interactions within or outside their institutions [5]. As mentioned by Ranard and colleagues [6], the use of “crowdsourcing can improve the quality, cost, and speed of a research project while engaging large segments of the public and creating novel science”. However, researchers are often reluctant to adopt crowdsourcing for creating, treating, and analyzing research data and there was little discussion on the difficulties associated with crowdsourcing research endeavors and how we might make progress in this area [7].

It is worth noting that there is a lack of systematic studies “investigating the applicability of crowdsourcing in not-for-profit fundamental research (as conducted in traditional universities)” [5]. In addition, few studies have already characterized how the synergies between mechanical and cognitive operators work and how to use them effectively for knowledge discovery and acquisition in scientific work scenarios [3]. Our aim is to describe how research might benefit from crowd computing based on literature found. While numerous areas of literature can illuminate this topic of inquiry, the actual contribution is closer to a survey paper and tries to explore the interplay between technology and the crowd in scientific settings.

In the ensuing section of this work, we try to revisit the theoretical background on crowd-computing hybrids by outlining in detail prior contributions. Section 3 explains the method followed for performing the systematic review presented here. Section 4 describes the main results of our study and discusses some challenges and open issues for further improvement. Section 5 provides some concluding remarks.

All science is a social system in its own nature, being characterized by challenges of massive scale [8]. As reported before, the collective wisdom of a crowd can be leveraged as a source of intellectual labor since “humans understand language, inference, implication, abstraction and concepts better than computers” [10]. However, harnessing crowdsourcing and human computation at large scale faces challenges that range from the difficult to scale up complexity and low quality responses [11] to the limited expertise or attention to cope with high-dimensional and ill-structured data [12], lack of motivation for participation [13], and worker honesty [14]. In the literature there are several examples of studies on crowdsourcing scientific tasks using Amazon Mechanical Turk (AMT)¹. For instance, Good et al. [15] recruited non-scientists to recognize disease concepts in biomedical paper abstracts. In addition, Brown and Allison [16] showed a high accuracy rate when using AMT to systematically evaluate scientific publications by distributing groups of Human Intelligence Tasks (HITs). Experiments on massive authorship of academic papers reported some challenges related to coordination mechanisms, tool design, content handling, and task differences [17]. There are also examples of leveraging an academic crowd for organizing conference sessions while extracting categories and clusters from high-dimensional data through crowd synthesis [12]. Some thematic reviews have already been performed to help identify parallels between crowd computing classes, descriptions, and systems while revealing gaps in the existing work as opportunities for new research (e.g., [19, 20, 21]).

As we enter an age of steadily larger and noisier data, a combination of both machine and human intelligence is required [22]. AI can help make the crowd more efficient and accurate through machine intelligence. On the other hand, “crowd intelligence can help train, supervise, and supplement automation” [23]. Most studies agree on the use of crowdsourcing as a reliable method for supervised and semi-supervised machine learning (e.g., active learning), from feature generation to prediction, deeper analysis, and classification of mass volumes of data [16]. Active learning and crowd-based human computation can be used to enhance the performance of automatic data classification and minimize the impact of possible erroneous or abusive feedback. Hybrid crowd-machine computation and mixed-initiative systems have been introduced as interactive, intelligent approaches that combine the strengths of human interaction with the algorithmic power of machine learning in order to solve problems that could not be solved by either computers or humans alone [24]. Examples of mixed-initiative systems in scientific discovery include PANDA [25] and Apolo [26]. Furthermore, Higgins [27] integrates automatic information extraction and human computation for crowdsourced knowledge acquisition. This kind of approach can be particularly fruitful in scientific contexts to refine machine-extracted metadata while providing evidence on demand using automatic classification techniques enabled by human crowd workers who can filter, process, and verify the information [3].

Systematic mapping is a process established on the identification, categorization, and analysis of scientific data concerning a certain research topic. The result is a structured summary that portrays the relationship between literature and categories [28]. SLR represents a critical part of research in evidence-based software engineering. Cruzes and Dybå [29] go even further by describing a SLR as “a concise summary of the best available evidence that uses explicit and rigorous methods to identify, critically appraise, and synthesize relevant studies on a particular topic”. The systematic review and mapping study described here follows published guidelines from works on software engineering (e.g., [30]). This section discusses the review protocol, formulate the research questions that the review intends to answer, and describes the strategy used to search for primary studies, study selection criteria and procedures, sources of studies, data extraction and synthesis strategies, and mapping procedures.

3.2 Search and Selection Processes

The authors performed searches on central scientific literature databases and followed the references in the resulting papers by means of a snowballing strategy for gathering new research studies that were then recorded in a spreadsheet. During the initial phase of the study, search engines (Google Scholar, ISI Web of Knowledge, Scopus) and traditional digital libraries (ACM Digital Library, IEEE Xplore, Springer Link, ScienceDirect, PLoS One, PubMed, BioMed Central, arXiv, etc.) were queried using a string sufficiently comprehensive for including research from multiple disciplines and research domains. Google Scholar was chosen as the primary search engine since it covers multivariate types of documents, while including papers from several fields of research. The following search strings were used to build the queries:

Open image in new window

Instead of searching for specific sources, which would not be so efficient due to the lack of perspectives spread across disciplines, reference lists of the publications found were recorded and examined towards identifying relevant studies. Furthermore, direct searches for related publications, prolific authors, and research groups were also performed. The search process was assessed for completeness by acquiring a large corpus of studies based on manual search of relevant sources. Snowballing enlarged the scope of this examination by considering aspects not previously addressed in the initial study. Thus, some limitations concerned with the use of a specific set of search terms, publication sources, and electronic databases were partially overtaken.

3.3 Inclusion/Exclusion Criteria and Primary Study Selection

The studies were screened according to various criteria pertinent to the research questions (Table 1). We established the following criteria for the inclusion of primary studies. Regarding a paper, it must be available as a full paper, written in English, and published in a peer-reviewed venue. Some exceptional documents (i.e., technical reports) were included due to the relevance of their content to the present study. As for the studies informed by the papers, they must report empirical evidence on conceptual dimensions behind crowd computing. The exclusion criteria consisted of eliminating duplicate studies that were not within the scope of this research. Publications that were clearly duplicated or for which we found newer and more complete versions (extensions) were excluded. In cases when articles present high levels of similarity, the most comprehensive study prevails. The database searches resulted in an extensive list of 3996 potential papers gathered after evaluation and deduplication.

Table 1.

Inclusion and exclusion criteria (adapted from Kitchenham [30])

Open image in new window

The title, abstract, and keywords were used to remove any studies not related to the research focus. From this sample, certain types of documents (e.g., theses and dissertations) were excluded. The SLR only included studies that were published between 2006 and 2017 and the key criterion required by a publication to be included was the relevance of the paper towards answering the research questions defined in this study. Afterwards, the remaining papers were read in order to remove any that do not fulfill the inclusion criteria. From the remaining entries, only 148 papers² were selected after full paper reading, where the initial and closing sections of each study were evaluated regarding their objectives. Journal and conference papers constitute the largest part of the sample, followed by workshop papers, book chapters, and symposium papers.

3.4 Data Extraction and Synthesis

The papers returned in the searching phase were stored using a data extraction form developed to gather all relevant data from the primary studies (Table 2). This registry supported the classification and analysis procedures. Only one researcher reviewed all papers and extracted metadata according to the data collection form for consistency.

Table 2.

Data extraction form (adapted from Cruzes and Dybå [29])

Open image in new window

A list of 1837 categories identified from primary studies was gathered and organized by their unique themes. Decisions on including or excluding features were made after reading the primary studies and the data were then grouped into meaningful clusters for synthesising qualitative evidence. Deduplication and aggregation were also performed in the clustering process, resulting in 158 themes/categories. Figure 1 summarizes all the steps followed in this review.

Open image in new window

In any review of the literature, placing the individual results within a larger framework is important to help build understanding of the larger pattern. A total of 8 clusters (#) emerged from the systematic review. A deeper insight into the key aspects and forms of crowdsourcing and massively collaborative science (RQ1) led us to explore distinguishing factors, dimensions and sample values as a starting point for academics who are interested in following up the extant literature on this topic. The time-space matrix (#1) was initially proposed by Johansen [31] to classify groupware by when participants are working at the same time (synchronous), same space (co-located), different time (asynchronous), or different places (remote). Schneider and colleagues [19] adopted the original version of this scheme to explain how time and space affect crowdware and crowd work settings. The contribution time (engagement profile) of each user is a relevant aspect of distributed human computation systems and crowd workers can be recruited and made available quickly [32]. As pointed out by Ponciano and co-workers [33], the level of engagement of each volunteer can be measured by relative activity duration, daily devoted time, and variation in periodicity. Uchoa et al. [34] go even further by claiming that “volunteers can assist researchers collecting and/or analyzing massive amounts of data that cover long periods of time or large geographic areas or employing some human cognitive ability in large scale”. As the authors put it, crowdware can reshape scientific work through crowd collaboration without temporal and spatial barriers.

Different forms of digital participation and public engagement (#2) can be established in scientific projects involving human crowds. For example, citizen science relies on getting ordinary citizens to voluntarily contribute toward scientific research [35]. Other forms of participation in crowdsourcing ecosystems include but are not limited to crowd funding (e.g., Experiment³), free and open source software development, altruistic crowdsourcing, and idea generation. Contributions can be individual or collaborative [20, 21]. In the latter case, we adopted the original Ellis et al.’s [36] 3C collaboration model to cluster the various ways and means of interacting by crowd members. Table 3 summarizes some evidence on the temporal/spatial issues and the participation modes in crowdsourcing and massively collaborative science.

Table 3.

Temporal/spatial issues and modes of digital participation in crowd science.

Open image in new window

Overlapping definitions describe a crowd (#3) as a large group of undefined, dispersed individuals showing varying work patterns, expertise, heterogeneity and performance, with little or no control imposed on them [37]. A crowd can read, classify and vote in a varied set of ways, providing supplementary observations and thus adding value to the findings obtained by researchers and small research teams [5]. As shown in the Table 4, a crowd is characterized by aspects like size, skills, social behavior, diversity, cultural differences, and motivation. Prior studies on quality control in crowdsourcing have addressed techniques like filtering out untrustworthy crowd workers for reducing bias [38]. Crowds are also represented by virtual proximity [39], social transparency [40], and social structure [23]. They can be hierarchy-neutral or hierarchical, being formed by strong/weak/absent ties [19]. Researchers’ roles vary in a crowdsourcing scenario. For instance, they can act as requestors, leaders (researchers design projects for which volunteers contribute data), collaborators, co-creators, and colleagues (volunteers that conduct research independently) [7].

Table 4.

Key characteristics of crowds.

Open image in new window

The success of a crowd-powered system is directly influenced by the involvement of a large number of contributors. It is worth noting that the motivation for participation (#4) is considered a central unit in almost all crowdsourcing studies. Motivating crowd members to contribute is complex by nature due to their individual and social differences [13]. The Self-Determination Theory [41] splits motivation constructs into two types: intrinsic/hedonistic (e.g., enjoyment) and extrinsic (e.g., payment). Sometimes, motivational factors overlap and are difficult to distinguish. As pointed out by Freitag and Pfeffer [42], many projects “have additional goals of engaging people in science and motivating them to incorporate scientific thought, hence the process of engaging citizen scientists can in itself also be a measure of success”. In a previous study with citizen scientists, Nov and colleagues [13] drew attention to the need for further analysis on the effects of motivation in scientific projects since the factors that improve participation may not lead to enhanced contribution quality. Table 5 summarizes some of the main motivational factors extracted from literature.

Table 5.

Motivational factors and reward schemes in crowdsourcing.

Open image in new window

Crowd work is usually decomposed into small tasks (#5) taking into account the particular needs and characteristics of each group of workers. Such independent and homogeneous tasks “may be structured through multistage workflows in which workers may collaborate either synchronously or asynchronously” using quality control mechanisms [23]. For instance, massively distributed authorship (e.g., writing an academic paper) involves a strong coordination effort to produce a high-quality output [17, 23]. On the other hand, crowd voting by means of critic reviews (e.g., Rotten Tomatoes⁴) comprises simple deskilled tasks (particular views/opinions). Such crowd-generated training data can be also incorporated into AI systems for informing future behavior. At the highest level, distributed scientific collaboration implicates several constraints on task design taking into account some aspects like training, supervision, and retention of crowd members for non-profit, scientific goals [35]. Table 6 presents some of the main issues related to crowdsouring tasks and process.

Table 6.

Crowdsourcing tasks and processes.

Open image in new window

To investigate RQ2, we analyzed papers discussing the functional attributes of crowd computing systems (#6). Crowdsourcing platforms are required for supporting interactions between crowd members and requestors while managing taks and outputs within an interactive working environment [23]. Hosseini et al. [21] provided a review of crowdsourcing platform features. According to the authors, a crowdsourcing system must be able to support enrolment, authentication, skill declaration, task assignment, assistance, result submission, coordination, supervision, and feedback loops when considering crowd-related interactions. Extrapolating the crowdsourcer domain, task broadcast and time/price negotiation must be also taken into account. Task-related features include result aggregation, history of completed tasks, quality and quantity threshold. Last but not least, a crowd-powered system must provide an interactive, ease to use interface while managing platform misuse and supporting payment, attraction, and interaction. The technology proficiency of participants [34] is another kind of aspect that must be also carefully considered when designing crowdsourcing systems for scientific purposes. Table 7 summarizes some crowdsourcing platform facilities identified in this literature review.

Table 7.

Crowdsourcing platform features (facilities).

Open image in new window

A closer inspection on the combination of both crowdsourcing and machine intelligence (RQ3) revealed a gap between automated reasoning and human cognition when performing complex tasks. Hybrid, crowd-machine interaction (#7) can close this gap by putting humans “into the loop” to overcome the failures of AI systems [22]. Nevertheless, some problems arise “when a disruptive shift like crowdsourcing crosses the traditional artificial boundaries we have constructed between knowledge areas” [18]. Reasoning abilities for hybrid intelligence allow better decisions towards the success of the collaborative activity [22]. Quinn and colleagues [9] addressed the different types of tradeoffs resulting from human labor, supervised learning, automated reasoning with human inputs, and mixed initiative systems. The authors presented a framework in which machines are leveraged by human-generated training data, while crowd workers can benefit from having automated reasoning results for which are necessary only simple actions for correcting them instead of doing all the work. It is also worth mentioning the introduction of hybrid algorithmic-crowdsourcing approaches for academic knowledge acquisition [25]. Table 8 presents some concepts related with crowd-computing hybrids with applicability for scientific research.

Table 8.

Crowd-computing hybrids and mixed-initiative approaches.

Open image in new window

The contextual settings within which crowd science may occur must be considered with caution [4]. Crowdsourcing schemes vary in the degree of control afforded to the crowd and the outputs provided by crowd members must be validated and aggregated in a coherent way. There are distinct forms of quality control and assurance in crowd-enabled settings [3]. Previous work has addressed the use of machine learning strategies for assessing worker quality while detecting lurkers, “spammers”, and other types of bad workers. Governance and management instruments are critical for crowdsourcing success. According to Hosseini and colleagues [21], “ethical issues are not fully and duly investigated in crowdsourcing activities”, and issues such as confidentiality, privacy and anonymity must be also considered when designing crowd-enabled systems. Table 9 shows general crowdsourcing aspects (#8) extracted from literature.

Table 9.

General dimensions of crowdsourcing.

Open image in new window

This paper presented a review of earlier contributions towards a reference model on the components that should be considered in crowd computing for data-driven scientific discovery. Such work shed a light to the theory and practice of innovative interactive systems, and the results achieved act as a foundation for more complex evaluation exercises to be undertaken. Scientific collaboration requires more than technology and there is little knowledge about theoretical frameworks for helping institutions and researchers analyzing concrete situations and identifying requirements before designing crowdsourcing systems. Crowd science can be particularly fruitful for making scientific work more accessible while enriching educational programs and disseminating results. A possible benefit of crowdsourcing is on a closer integration between human and machine intelligence and we need to deal with the question of what parts of scientific work to crowdsource and how to support these processes with AI. Putting AI on guiding (and be guided by) crowds enlarges the design space for application developers [23] and there is a large path of further improvement towards hybrid classifiers embedding crowds inside of machine learning architectures.