A Label Completion Approach to Crowd Approximation

  • Toshihiro Watanabe
  • Hisashi Kashima
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8835)


Majority vote is one of the most common methods for crowdsourced label aggregation to get higher-quality labels. In this paper, we extend the work of Donmez et al. that estimates majority labels with a small subset of crowdsourcing workers in order to reduce financial and time costs. Our proposed method estimates the majority labels more accurately by completing missing labels to approximate the whole crowds even if some workers do not answer labels. Experimental results show that the proposed method approximates crowds more accurately than the method without label completion.


Recommendation System Majority Vote Computer Support Cooperative Work Neural Information Processing System Probabilistic Matrix Factorization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Machine Learning 47(2-3), 235–256 (2002)CrossRefzbMATHGoogle Scholar
  2. 2.
    Donmez, P., Carbonell, J.G., Schneider, J.: Efficiently learning the accuracy of labeling sources for selective sampling. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259–268 (2009)Google Scholar
  3. 3.
    Ertekin, S., Hirsh, H., Rudin, C.: Learning to predict the wisdom of crowds. In: Proceedings of Collective Intelligence (2012)Google Scholar
  4. 4.
    Jung, H.J., Lease, M.: Improving quality of crowdsourced labels via probabilistic matrix factorization. In: Proceedings of the 4th Human Computation Workshop at AAAI, pp. 101–106 (2012)Google Scholar
  5. 5.
    Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Reserch and Development in Information Retrieval, pp. 3–12 (1994)Google Scholar
  6. 6.
    Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: Grouplens: an open architecture for collaborative filering of netnews. In: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, pp. 175–186 (1994)Google Scholar
  7. 7.
    Salakhutdinov, R., Mnih, A.: Probabilistic matrix factorization. In: Advances in Neural Information Processing Systems (NIPS), vol. 20, pp. 1257–1264 (2007)Google Scholar
  8. 8.
    Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 254–263 (2008)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Toshihiro Watanabe
    • 1
  • Hisashi Kashima
    • 2
  1. 1.The University of TokyoTokyoJapan
  2. 2.Kyoto UniversityKyotoJapan

Personalised recommendations