Statistics and Computing

, Volume 20, Issue 3, pp 317–334 | Cite as

Semi-parametric analysis of multi-rater data

  • Simon Rogers
  • Mark Girolami
  • Tamara Polajnar


Datasets that are subjectively labeled by a number of experts are becoming more common in tasks such as biological text annotation where class definitions are necessarily somewhat subjective. Standard classification and regression models are not suited to multiple labels and typically a pre-processing step (normally assigning the majority class) is performed. We propose Bayesian models for classification and ordinal regression that naturally incorporate multiple expert opinions in defining predictive distributions. The models make use of Gaussian process priors, resulting in great flexibility and particular suitability to text based problems where the number of covariates can be far greater than the number of data instances. We show that using all labels rather than just the majority improves performance on a recent biological dataset.


Semi-parametric Gaussian processes Machine learning Multi-rater Classification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Albert, J., Chib, S.: Sequential ordinal modeling with applications to survival data. Biometrics 57, 829–836 (2001) CrossRefMathSciNetGoogle Scholar
  2. Albert, J.H., Chib, S.: Bayesian analysis of binary and polychotomous response data. J. Am. Stat. Assoc. 88(422), 669–679 (1993) zbMATHCrossRefMathSciNetGoogle Scholar
  3. Bickel, S., Brefeld, U., Faulstich, L., Hakenberg, J., Leser, U., Plake, C., Scheffer, T.: A support vector machine classifier for gene name recognition. In: EMBO Workshop: A Critical Assessment of Text Mining Methods in Molecular Biology, Granada, Spain, March 2004 Google Scholar
  4. Chu, W., Ghahramani, Z.: Gaussian processes for ordinal regression. J. Mach. Learn. Res. 6, 1–48 (2005) MathSciNetGoogle Scholar
  5. Cohen, K., Fox, L., Ogren, P., Hunter, L.: Corpus design for biomedical natural language processing. In: Proceedings of the ACL-ISMB Workshop on Linking Biological Literature, Ontologies and Databases (Jan. 2005) Google Scholar
  6. Cowles, M.K.: Accelerating Monte Carlo Markov Chain convergence for cumulative-link generalized linear models. Stat. Comput. 6, 101–111 (1996) CrossRefGoogle Scholar
  7. Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the em algorithm. Appl. Stat. 28(1), 20–28 (1979) CrossRefGoogle Scholar
  8. Gelman, A., Carlin, J., Stern, H., Rubin, D.: Bayesian Data Analysis. Chapman&Hall, London (2004) zbMATHGoogle Scholar
  9. Girolami, M., Rogers, S.: Variational Bayesian multinomial probit regression with Gaussian process priors. Neural Comput. 18(8), 1790–1817 (2006). doi: 10.1162/neco.2006.18.8.1790 zbMATHCrossRefMathSciNetGoogle Scholar
  10. Girolami, M., Zhong, M.: Data integration for classification problems emplying Gaussian process priors. Adv. Neural Inf. Process. Syst. 21 (2007) Google Scholar
  11. Johnson, V.: An alternative to traditional GPA for evaluating student performance. Stat. Sci. 12(4), 251–269 (1997) CrossRefGoogle Scholar
  12. Johnson, V., Albert, J.: Ordinal Data Modeling. (Jan. 1999)
  13. Johnson, V.E.: On Bayesian analysis of multirater ordinal data: An application to automated essay grading. J. Am. Stat. Assoc. 91(433), 42–51 (1996) zbMATHCrossRefGoogle Scholar
  14. Rogers, S., Girolami, M.: Multi-class semi-supervised learning with the ε-truncated multinomial probit Gaussian process. J. Mach. Learn. Res. Workshop Conf. Proc. 1, 17–32 (2007) Google Scholar
  15. Smyth, P., Fayyad, U., Burl, M., Perona, P., Baldi, P.: Inferring ground truth from subjective labelling of venus images. Adv. Neural Inf. Process. Syst. 7 (1995) Google Scholar
  16. Uebersax, J.S.: Statistical modeling of expert ratings on medical treatment appropriateness. J. Am. Stat. Assoc. 88(422), 421–427 (1993) CrossRefGoogle Scholar
  17. Versley, Y.: Disagreement dissected: Vagueness as a source of ambiguity in nominal (co-) reference. In: Ambiguity in Anaphora Workshop Proceedings (2006) Google Scholar
  18. Wilbur, W.J., Rzhetsky, A., Shatkay, H.: New directions in biomedical text annotation: definitions, guidelines and corpus construction. BMC Bioinf. 7, 356–356 (2006) CrossRefGoogle Scholar
  19. Williams, C.K., Barber, D.: Bayesian classification with Gaussian processes. IEEE Trans. Pattern Anal. Mach. Intell. 20(12), 1342–1351 (1998) CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Department of Computing ScienceUniversity of GlasgowGlasgowUK

Personalised recommendations