SciCrowd: Towards a Hybrid, Crowd-Computing System for Supporting Research Groups in Academic Settings

A major research path in today’s computing landscape has been focused on analyzing datasets whose volume, velocity, and variety are so extreme that the current automatic tools are inadequate for their accurate collection, management, and analysis in a reasonable amount of time [17]. As the literature increases at an exponential rate, scholars need to adapt their institutions and practices by improving searching, analytical, and filtering mechanisms to select the most relevant data sources. Nonetheless, manual labor applied on examining variances, correlating evidence, and compiling descriptive statistics is an exhaustive and subjective process [13]. At the forefront are the rapidly advancing techniques of AI such as machine learning running on large collections of data [1]. However, computer algorithms involve expensive training being prone to errors. Dunne and colleagues [14] go even further by arguing that the “current theories and tools are directed at finding a paper or website, not gaining an understanding of the key papers, authors, controversies, and hypotheses.” In this sense, research scholars interested in answering a particular question need intelligent systems robust enough to assist them in exploring always-evolving knowledge representations as a “science of science measurement” [19].

In this work, SciCrowd is introduced as a crowd-powered system prototype under development “from the ground up” towards an open participation model in which a large amount of researchers and crowd workers/volunteers can contribute for improving the way as research data are processed. We assume that crowd-computing hybrids constitute a valuable instrument to address some of the limitations of current applications by enabling us to identify new relations between research topics, authors, and groups [10]. Meaningful categories, filters, and summaries can be generated through mixed-initiative approaches combining human and machine intelligence [13]. Such synergies between mechanical and cognitive operators (the “crowd”) can contribute to create open knowledge bases supported by collaborative participation.

The paper is organized as follows. Section 2 provides an overview of key issues in the form of theoretical bridge between crowd and machine intelligence for enhancing research pursuits. Section 3 presents a brief conceptualization of the requirements for SciCrowd as an experimental human-centered hybrid system for scientific discovery. The implementation details are also discussed with emphasis on the SciCrowd’s prototype and architecture. This section also provides some exploratory remarks on the results achieved from a survey with crowdsourcing researchers. The paper finishes discussing limitations and future directions that need consideration in the deployment of a crowd-powered, self-organizing system for scientific discovery.

Scientific knowledge production is being increasingly dependent on collaborative efforts and new forms of collaboration and data analysis are dawning as result of social-technical advances achieved in e-Science through CSCW research [2]. Various approaches have been proposed to leverage crowd work for improving the quality of data. Human crowds can combine Human Intelligence Tasks (HITs) and large-scale database systems such as MTurk¹ to cope with the ambiguity, vagueness, heterogeneity, scale, timeliness, and complexity of scientific data [10]. Empirical studies have emphasized crowdsourcing efforts in domains such as taxonomy development [22], machine learning experiments using MTurk [20], and scientific discovery games (e.g., Foldit²) [6]. In addition, several crowd-powered systems have been presented in the literature. For instance, GATE Crowdsourcing Plugin [12] offers infrastructural support for supporting a massive pool of users on mapping corpora to microtasks while automatically generating crowdsourcing interfaces for classification and annotation. CrowdScape [21] measures crowd behavior and outputs though interactive visualization and mixed-initiative machine learning. Moreover, Apolo [8] allows interactive large graph sensemaking by combining machine learning and visualization. On the other hand, PANDA [9] is a hybrid algorithmic-crowdsourcing platform for academic knowledge discovery.

Mixed-initiative systems have been introduced as interactive, intelligent approaches that combine the strengths of human interaction with the algorithmic power of AI to solve problems that could not be solved by either computers or humans alone [11]. In this kind of systems, human work is both used directly to complete HITs and as feedback for machine learning classifiers training to complete that work in the future. Such hybrid approach can be particularly fruitful in crowd science projects to refine machine-extracted metadata while providing evidence on demand by using automatic classification techniques enabled by human crowd workers who can process, filter, classify, and verify the information [10].

A paradigmatic shift in information systems design that can be taken into account in the design and development of a crowd-computing system for scientific discovery involves the consideration of IT artifacts and social practices as central units of research [15]. Theoretically grounded from a design science perspective, such approach includes goal-directed collective activities, routines, and resources to create explicit knowledge by means of reflection and concept formation (e.g., linguistic signs, organizational schemes, technical artifacts) as socially situated practices.

Research tasks such as performing a systematic literature review involve a set of laborious, time-consuming, and sometimes wasteful efforts. This includes the identification of relevant sources, data extraction, and quality assessment [5]. In this scenario, relevant sources and data dimensions can be ignored during the process. Thus, suitable crowd-powered systems are required to produce, share, filter, and combine scientific discoveries. In the literature, crowdsourcing has been closely related to processes, task characteristics, and participants [10]. Tinati and colleagues [7] identified numerous factors when designing citizen science platforms, including task specificity, community development, task design, and engagement. The authors pointed out to the likelihood of using periodic feedback, roles and privileges, and task context as motivational factors. In addition, the environment must be easy to use, fast, intuitive, and reliable. Crowd aspects (including size, type, social structure, behavior), quality control, and temporal issues (e.g., daily devoted time) assume particular importance on the design of crowd-powered systems [10]. Hybrid classifiers must be trained according to the crowd behavior and the system must enable the crowd to selectively supplement weak regions of the classifier [16].

3.1 Baseline System Design and Prototyping

SciCrowd is a system prototype based on a crowd-enabled model [4] that supports the entire process of research data analysis. In this approach, crowd members are exposed to situations involving human factors like expertise and motivation. HITs are created by users and associated to collaboration features (e.g., annotation). Our ultimate goal is to build a mixed-initiative interface for (and by) researchers and citizen scientists. Figure 1 presents the system prototype towards the design of a web-based interface for extracting facts from publication records through human-machine interaction. In this scenario, a contributor can be aware of new papers in the system or insert a new entry, see who is online in the system, edit publication metadata, set conceptual units and descriptions, and getting access to data categorized by machine.

Open image in new window

Currently, the prototype comprises a limited set of features that range from edit a publication record automatically extracted from DBLP³ or added manually by users to the settings panel (e.g., group roles) and main page. Publication details can be visualized pressing the “show” button, which opens an internal page that allows to insert data about a paper. For instance, consider the following scenario. A paper on “medical informatics” could be classified by subarea (e.g., cognitive aging), method, setting, key concepts, findings, and social-technical aspects concerning a certain technology (e.g., a Wiki intended to support knowledge exchange in public health). All these data could be filtered considering specific research purposes. For example, a user could be interested on understanding relations among concepts and related areas in machine learning for medical purposes and/or identifying features in health care systems implemented by Portuguese researchers in a certain period.

Our approach to extract information from public data sources is based on a previous proposal for a hybrid human-computer scientific information extraction pipeline [23]. As initially addressed by Correia and colleagues [18] and followed in further proposals [9], the preprocessing stage includes the automatic extraction of metadata from each paper using DBLP. To test the effectiveness of our information extraction algorithm, metadata from papers were indexed using a structure that comes from two bibliographic reference management systems (EndNote, BibTeX). Currently, the database contains data about 28,991,154 papers. How scientific data are indexed, classified, and visualized differs from field to field through an architecture of participation that gain value as more users cooperate [2]. The system allows the creation of user groups with predefined permissions (e.g., edit) by using an authentication mechanism. An innovative feature is being planned for extracting altmetrics [3], including download data and social mentions by country, discipline, and professional status. Figure 2 provides an overview of the current system architecture.

Open image in new window

This paper emphasizes the implications for design concerning a crowd-enabled, self-organizing system under development towards achieving a higher level of engagement by researchers and citizen scientists. SciCrowd system’s prototype aims to improve the way as publication records are evaluated while harnessing the wisdom of a crowd. The main target relies on improving the effective use of data to solve complex problems at large-scale, and the possibility to adapt this tool to multivariate scientific fields. The crowd can be particularly helpful in this process by indexing metadata and fixing errors while providing new interpretations on the published data through highly detailed and filtered data granules. The outputs can be established on understanding how research communities form and evolve over time. Further steps will be focused on providing features for supporting hybrid reasoning using AI inferring on contributions from crowd members performing HITs such as systematic literature reviews and scientometric studies. Despite the prospects for research in crowd science, we must be able to consider incentives, ethical constraints, and social-technical aspects for “closing the loop” in developing crowd-powered scientific computing systems.