Learning from Multiple Annotators with Gaussian Processes

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6792)


In many supervised learning tasks it can be costly or infeasible to obtain objective, reliable labels. We may, however, be able to obtain a large number of subjective, possibly noisy, labels from multiple annotators. Typically, annotators have different levels of expertise (i.e., novice, expert) and there is considerable diagreement among annotators. We present a Gaussian process (GP) approach to regression with multiple labels but no absolute gold standard. The GP framework provides a principled non-parametric framework that can automatically estimate the reliability of individual annotators from data without the need of prior knowledge. Experimental results show that the proposed GP multi-annotator model outperforms models that either average the training data or weigh individually learned single-annotator models.


Gaussian Process Predictive Equation Gaussian Process Regression Gaussian Process Model Supervise Learning Task 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Raykar, V.C., Yu, S., Zhao, L.H., Jerebko, A., Florin, C.: Supervised learning from multiple experts: whom to trust when everyone lies a bit. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 889–896 (2009)Google Scholar
  2. 2.
    Raykar, V.C., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L.: Learning from crowds. JMLR 11, 1297–1322 (2010)MathSciNetGoogle Scholar
  3. 3.
    Smyth, P., Fayyad, U., Burl, M., Perona, P., Baldi, P.: Inferring ground truth from subjective labelling of venus images. In: Advances in Neural Information Processing Systems, vol. 7, pp. 1085–1092 (1995)Google Scholar
  4. 4.
    Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceeding of the 14th ACM SIGKDD International Conference On Knowledge Discovery and Data Mining, pp. 614–622 (2008)Google Scholar
  5. 5.
    Snow, R., O’Connor, B., Jrafsky, D., Ng, A.: Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 254–263 (2008)Google Scholar
  6. 6.
    Sorokin, A., Forsyth, D.: Utility data annotation with Amazon Mechanical Turk. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8 (2008)Google Scholar
  7. 7.
    Warfield, S.K., Zou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (STAPLE). An algorithm for the validation of image segmentation. IEEE Trans. Med. Imag. 23(7), 903–921 (2004)CrossRefGoogle Scholar
  8. 8.
    Cholleti, S.R., Goldman, S.A., Blum, A., Politte, D.G., Don, S.: Veritas: combining expert opinions without labeled data. In: 20th IEEE International Conference on Tools with Artificial Inteligence (2008)Google Scholar
  9. 9.
    Ristovski, K., Das, D., Ouzienko, V., Guo, Y., Obradovic, Z.: Regression Learning with Multiple Noisy Oracles. In: 19th European Conference on Artificial Intelligence, pp. 445–450 (2010)Google Scholar
  10. 10.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian processes for machine learning. MIT Press, Cambridge (2006)Google Scholar
  11. 11.
    Dekel, O., Shamir, O.: Good Learners for Evil Teachers In: Proceedings of the 26th International Conference on Machine Learning, pp. 233–240 (2009)Google Scholar
  12. 12.
    Dekel, O., Shamir, O.: Vox populi: Collecting high-quality labels from a crowd. In: Proceedings of the 22nd Annual Conference on Learning Theory, pp. 377–386 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  1. 1.Dept. of Electrical Engineering - Control SystemsTechnical University EindhovenEindhovenThe Netherlands
  2. 2.Intelligent SystemsRadboud University NijmegenNijmegenThe Netherlands

Personalised recommendations