Low-Cost Preference Judgment via Ties

  • Kai Hui
  • Klaus Berberich
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10193)


Preference judgment, as an alternative to graded judgment, leads to more accurate labels and avoids the need to define relevance levels. However, it also requires a larger number of judgments. Prior research has successfully reduced that number to \(\mathcal {O}(N_d\,\log {N_d})\) for \(N_d\) documents by assuming transitivity, which is still too expensive in practice. In this work, by analytically deriving the number of judgments and by empirically simulating the ground-truth ranking of documents from Trec Web Track, we demonstrate that the number of judgments can be dramatically reduced when allowing for ties.


  1. 1.
    Ailon, N., Mohri, M.: An efficient reduction of ranking to classification. arXiv 2007 (2007)Google Scholar
  2. 2.
    Carterette, B., Bennett, P.N.: Evaluation measures for preference judgments. In: SIGIR 2008 (2008)Google Scholar
  3. 3.
    Carterette, B., Bennett, P.N., Chickering, D.M., Dumais, S.T.: Here or there: preference judgments for relevance. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 16–27. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-78646-7_5 CrossRefGoogle Scholar
  4. 4.
    Cormen, T.H.: Introduction to Algorithms. MIT Press, Cambridge (2009)zbMATHGoogle Scholar
  5. 5.
    Kazai, G., Yilmaz, E., Craswell, N., Tahaghoghi, S.M.: User intent and assessor disagreement in web search evaluation. In: CIKM 2013 (2013)Google Scholar
  6. 6.
    Niu, S., Guo, J., Lan, Y., Cheng, X.: Top-k learning to rank: labeling, ranking and evaluation. In: SIGIR 2012 (2012)Google Scholar
  7. 7.
    Radinsky, K., Ailon, N.: Ranking from pairs and triplets: information quality, evaluation methods and query complexity. In: WSDM 2011 (2011)Google Scholar
  8. 8.
    Song, R., Guo, Q., Zhang, R., Xin, G., Wen, J.-R., Yu, Y., Hon, H.-W.: Select-the-best-ones: a new way to judge relative relevance. Inf. Process. Manag. 47(1), 37–52 (2011)CrossRefGoogle Scholar
  9. 9.
    Wang, J., Li, G., Kraska, T., Franklin, M.J., Feng, J.: Leveraging transitive relations for crowdsourced joins. In: SIGMOD 2013 (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Max Planck Institute for InformaticsSaarbrückenGermany
  2. 2.Saarbrücken Graduate School of Computer ScienceSaarbrückenGermany
  3. 3.htw saarSaarbrückenGermany

Personalised recommendations