A Label Completion Approach to Crowd Approximation
Majority vote is one of the most common methods for crowdsourced label aggregation to get higher-quality labels. In this paper, we extend the work of Donmez et al. that estimates majority labels with a small subset of crowdsourcing workers in order to reduce financial and time costs. Our proposed method estimates the majority labels more accurately by completing missing labels to approximate the whole crowds even if some workers do not answer labels. Experimental results show that the proposed method approximates crowds more accurately than the method without label completion.
KeywordsRecommendation System Majority Vote Computer Support Cooperative Work Neural Information Processing System Probabilistic Matrix Factorization
Unable to display preview. Download preview PDF.
- 2.Donmez, P., Carbonell, J.G., Schneider, J.: Efficiently learning the accuracy of labeling sources for selective sampling. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259–268 (2009)Google Scholar
- 3.Ertekin, S., Hirsh, H., Rudin, C.: Learning to predict the wisdom of crowds. In: Proceedings of Collective Intelligence (2012)Google Scholar
- 4.Jung, H.J., Lease, M.: Improving quality of crowdsourced labels via probabilistic matrix factorization. In: Proceedings of the 4th Human Computation Workshop at AAAI, pp. 101–106 (2012)Google Scholar
- 5.Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Reserch and Development in Information Retrieval, pp. 3–12 (1994)Google Scholar
- 6.Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: Grouplens: an open architecture for collaborative filering of netnews. In: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, pp. 175–186 (1994)Google Scholar
- 7.Salakhutdinov, R., Mnih, A.: Probabilistic matrix factorization. In: Advances in Neural Information Processing Systems (NIPS), vol. 20, pp. 1257–1264 (2007)Google Scholar
- 8.Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 254–263 (2008)Google Scholar