Answer validation for generic crowdsourcing tasks with minimal efforts

Abstract

Crowdsourcing has been established as an essential means to scale human computation in diverse Web applications, reaching from data integration to information retrieval. Yet, crowd workers have wide-ranging levels of expertise. Large worker populations are heterogeneous and comprise a significant amount of faulty workers. As a consequence, quality insurance for crowd answers is commonly seen as the Achilles heel of crowdsourcing. Although various techniques for quality control have been proposed in recent years, a post-processing phase in which crowd answers are validated is still required. Such validation, however, is typically conducted by experts, whose availability is limited and whose work incurs comparatively high costs. This work aims at guiding an expert in the validation of crowd answers. We present a probabilistic model that helps to identify the most beneficial validation questions in terms of both improvement in result correctness and detection of faulty workers. By seeking expert feedback on the most problematic cases, we are able to obtain a set of high-quality answers, even if the expert does not validate the complete answer set. Our approach is applicable for a broad range of crowdsourcing tasks, including classification and counting. Our comprehensive evaluation using both real-world and synthetic datasets demonstrates that our techniques save up to 60% of expert efforts compared to baseline methods when striving for perfect result correctness. In absolute terms, for most cases, we achieve close to perfect correctness after expert input has been sought for only 15% of the crowdsourcing tasks.

Appendices

A Further details on datasets

1.1 Real-world data and task design

We have used real-world datasets from different domains, namely people (ppl),object (obj), product reviews (prod), argument (arg), and bluebird (bb). Opting for a generic crowdsourcing setting, our task design uses the default multiple-choice question template from AMT [23, 32]. Further complex, yet out of scope, task designs aiming for human factors and exploiting domain-specific knowledge can be found in [34]. In the ppl dataset, workers have to count the number of people in an real-life image. The crowdsourcing tasks of the obj dataset comprise counting the number of people in digital-art picture. However, the questions of the obj dataset are more difficult than the questions of the ppl dataset as the people in digital-art picture are harder to recognize. In the prod dataset, workers are asked to annotate whether a review expresses positive, neutral or negative meaning. The tasks for the arg dataset require the crowd workers to extract claim and evidence related to a topic from articles from the web. In the bb dataset, workers have to identify one of two types of birds in an image. The similarity function—input of our model—are simply computed by uniformly normalizing the labels into natural number space. The ground truth/expert validation is provided by experts in the field.

1.2 Synthetic data

We used several generated datasets. Since this data should exhibit similar characteristics as real-world data, we considered several parameters for the data generation, in particular: (i) n—the number of objects, (ii) k—the number of workers, (iii) m—the number of labels, (iv) r—the reliability of normal workers, reflecting the probability of their answers being correct, (v) $\sigma$—the percentage of spammers in the worker population and (vi) sim—the similarity between labels, simulated as a uniform distribution in [0, 1]. For the synthetic dataset, we also simulated the ground truth (the correct labels) for the objects. However, it is not known by the simulated workers and only used to simulate the answer validations.

An important part of our synthetic data is the crowd simulation. We follow a practical guideline [22] to simulate the different worker characteristics of the crowd. Specially, we distribute the worker population into $\alpha \%$ reliable workers, $\beta \%$ sloppy workers and $\gamma \%$ spammers. According to a study on crowd population at real-world crowdsourcing services [31], we assign the default values of these parameters as follows: $\alpha = 43$, $\beta =32$ and $\gamma = 25$. In the experiments, the distribution of the worker types is the same as discussed unless stated otherwise.

B Evaluations of expert guidance (cont’d)

Fig. 15

Effect of number of workers

Fig. 16

Effect of worker reliability

In the following experiments, we analyze the effects of the guiding strategy with different crowdsourcing setup, including the number of labels, the number of workers, worker reliability, question difficulty, and the presence of spammers. Since these experiments (except the experiment on question difficulty) require changing the workers’ characteristics (which is not known for the real-world datasets), they are conducted using synthetic data.

We compare the results obtained with our guiding approach (hybrid) to a baseline guiding method that selects the object with the highest uncertainty to seek feedback (baseline):

$$\begin{aligned} \mathrm{select}(O) = \mathop {{{\mathrm{arg\,max}}}}\limits _{o \in O} H(o) \end{aligned}$$

Our hybrid approach is different from the baseline as it further considers the consequences of validation in addition to the mutually reinforcing relations between the reliability of workers and assignment correctness.

1.1 Effect of the number of workers

The idea behind crowdsourcing is that individual crowd answers complement each other. Thus, the aggregation of answers should be closer to the truth as more workers participate [67]. To evaluate the effect of the number of workers on the performance of our approach, we rely on a synthetic dataset containing 50 objects. We vary the number of workers k from 20 to 40 that assign one of three labels to the objects. Figure 15 illustrates an important finding that our approach leads to better results for any number of workers. Taking a fixed amount of expert input, precision increases if more workers are employed. The reason is the widely quoted “wisdom of the crowd” [67], which eventually leads to better precision. Another finding is that the precision improvement with the same amount of expert input is higher if we have more workers (most right plot in Fig. 15). This is expected since, by having more workers, we acquire more answers for the same question, which results in better estimates of assignment probabilities and worker reliabilities. Our approach, thus, has a higher chance to select the objects that lead to a large gain in correctness.

In sum, the two findings suggest that increasing the number of workers is beneficial not only for computing assignment probabilities, but also for guiding answer validation. For the remaining experiments, we fix the number of workers to be the smallest tested value ($k=20$), which is the most challenging scenario.

Fig. 17

Effect of spammers

1.2 Effect of worker reliability

We further explored the effects of the worker reliability r on the effectiveness of our approach. As above, we used a dataset of 20 workers assigning one out of three labels to 50 objects. We then varied the reliability of the non-spammer workers from 0.65 to 0.75.

Figure 16 illustrates a significant improvement in precision using our approach (hybrid) compared to the baseline method. For instance, if the average worker reliability is 0.7, to achieve a precision of 0.95, our approach requires expert input for 20% of the objects, whereas the baseline method requires input for 50% of the objects. In other words, the amount of efforts the baseline method requires is 2.5 times that of our approach. Also, with the same amount of feedback, precision is increased if the average reliability of the workers is higher (most right plot in Fig. 16). This is because an answer set provided by reliable workers requires less validation than an answer set coming from unreliable workers.

1.3 Effect of spammers

In this experiment, we studied the robustness of our guiding approach to spammers using the same dataset as in the previous experiment (20 workers, three labels, 50 objects). We varied the percentage of spammers $\sigma$ in the worker population from 15 to $35\%$ to analyze the effect of these spammers.

Independent of the percentage of spammers, our approach (hybrid) outperforms the baseline method, see Fig. 17. The largest difference between the two approaches is observed when the percentage of spammers is 15%. In that case, to achieve a precision of 0.95, our approach needs 20% of expert input, while the baseline method requires 50%. Regarding the precision improvement (right most plot in Fig. 17), the results are relatively similar across different percentages of spammers. For instance, using 50% of expert input, we are able to increase the precision of the deterministic assignment by 80%, independent of the percentage of spammers. Hence, our approach is indeed robust to the presence of spammers.

1.4 Effects of question difficulty

Beside worker reliability, another factor that can affect the performance of our method is the question difficulty. For hard questions, even reliable workers may give incorrect answers. As a result, there is a need to analyze the effects of question difficulty on the performance of our approach. We compared our approach with the baseline approach using two datasets: ppl and obj, where the questions in the obj dataset is harder than the other. The experimental results are shown in Fig. 18, where the x-axis depicts the expert efforts while the y-axis illustrates the precision of the deterministic assignment.

We observe that our approach is able to outperform the baseline approach for both datasets, meaning that the approach is robust against question difficulty. For instance, for the ppl dataset with easy question, our approach needs only 20% of expert effort to achieve a precision of 0.95 while the baseline approach needs over 60% of expert efforts. Also, the performance of our approach when the questions are easy is better than in the setup with hard questions. This is expected and can be explained as follows. In the dataset with easy questions, most of the workers are able to give the correct answers, which makes the uncertainty in the dataset low. As a result, with the same amount of feedbacks, we can improve the precision higher than when the questions are hard.

C Cost trade-offs (cont’d)

We complement the experiments reported in Sect. 7.9 by studying the effects of question difficulty, spammers, and worker reliability when comparing the EV approach with the WO approach.

Fig. 18

Effects of question difficulty

1.1 Effects of question difficulty

In this experiment, we compare our EV approach with the WO approach with respect to the difficulty of the questions. We remove the answers from the answer matrix randomly such that 13 answers remain per question ($\phi _0 = 13$). Then, to simulate the addition of answers for the WO approach, we add the answers back to the questions. We fix the expert-crowd cost ratio to $\theta =25$ and average the results over 100 experiment runs.

Fig. 19

Effect of question difficulty on cost

The experimental results are shown in Fig. 19 where the X-axis depicts the normalized cost and the Y-axis measures the precision improvement of the deterministic assignment. The precision improvement of the EV approach is always higher than that of the WO approach, indicating that our EV approach is robust against the effects of question difficulty.

1.2 Effects of spammers

In this experiment, we analyze the effects of spammers by varying the percentage of spammers in the dataset from 15 to 35%. The experiment is conducted on the synthetic dataset with $\phi _0 = 13$, $\theta =25$.

Fig. 20

Effects of spammers on cost

The results illustrated in Fig. 20 show the benefits of using our approach with different percentages of spammers. The EV approach is able to achieve high precision improvement with a small amount of cost. For instance, when $\sigma =35\%$, to improve the precision by 80%, a cost of 30 is required for the EV approach while the WO approach needs twice the amount. Also, the more spammers are part of the population, the better becomes the performance of the EV approach regarding the WO approach. For example, the difference in cost to achieve 80% precision improvement is about 15 when the percentage of spammers is 15%, but this increases three times to 30 as the percentage of spammers increases to 35%. Again, the reason is that as the percentage of spammer increases, the WO suffers from adding more answers as they are more likely to come from unreliable workers.

1.3 Effects of worker reliability

Worker reliability can affect the quality of crowd answers, thus also affects the cost model. If the worker reliability is high, the expert can spend less effort to give feedbacks as most of the answers are already correct. On the other hand, when the worker reliability is low, more feedbacks from the expert are required to achieve the same amount of precision. In this experiment, we analyze the effects of worker reliability on the cost of validating the crowd answers by varying the reliability of the normal workers from 0.6 to 0.7. Similar to the above experiment, we fix the following parameter: $\phi _0 = 13$, $\theta =25$ and the workers population is simulated as discussed in Sect. 7.1.

Fig. 21

Effects of worker reliability on cost

The obtained results are illustrated in Fig. 21, which highlights the relation between the cost normalized over each question and the precision of the deterministic assignment. Interestingly, when the reliability of the workers is 0.6, the precision of the deterministic assignment using the WO approach converges to 0 as we add more answers. The reason is that as we decrease the worker reliability, the average worker reliability becomes less than 0.5, which makes the precision converge to 0. This shows that adding more answers to the answer set may not improve but reduce the quality due to unreliable workers. When the reliability of the workers is 0.65, the precision of the deterministic assignment using the WO approach improves very slowly as the average reliability of the whole population is about 0.5. On the other hand, when the reliability of the workers is 0.7, the precision of the WO approach converges to 1. Yet, it requires higher cost to reach the same amount of precision as the EV approach. In summary, this experiment shows that our approach is robust against the reliability of the workers.