Here or There

Preference Judgments for Relevance
  • Ben Carterette
  • Paul N. Bennett
  • David Maxwell Chickering
  • Susan T. Dumais
Conference paper

DOI: 10.1007/978-3-540-78646-7_5

Part of the Lecture Notes in Computer Science book series (LNCS, volume 4956)
Cite this paper as:
Carterette B., Bennett P.N., Chickering D.M., Dumais S.T. (2008) Here or There. In: Macdonald C., Ounis I., Plachouras V., Ruthven I., White R.W. (eds) Advances in Information Retrieval. ECIR 2008. Lecture Notes in Computer Science, vol 4956. Springer, Berlin, Heidelberg

Abstract

Information retrieval systems have traditionally been evaluated over absolute judgments of relevance: each document is judged for relevance on its own, independent of other documents that may be on topic. We hypothesize that preference judgments of the form “document A is more relevant than document B” are easier for assessors to make than absolute judgments, and provide evidence for our hypothesis through a study with assessors. We then investigate methods to evaluate search engines using preference judgments. Furthermore, we show that by using inferences and clever selection of pairs to judge, we need not compare all pairs of documents in order to apply evaluation methods.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Ben Carterette
    • 1
  • Paul N. Bennett
    • 2
  • David Maxwell Chickering
    • 3
  • Susan T. Dumais
    • 2
  1. 1.University of Massachusetts Amherst 
  2. 2.Microsoft Research 
  3. 3.Microsoft Live Labs 

Personalised recommendations