Judging Relevance Using Magnitude Estimation
Part of the
Lecture Notes in Computer Science
book series (LNCS, volume 9022)
Magnitude estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if magnitude estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) magnitude estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable magnitude estimation scores.
KeywordsMagnitude Estimation Ordinal Scale Expert Judgement Relevance Level Relevance Score
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Eisenberg, M.: Measuring relevance judgements. Information Processing and Management 24, 373–389 (1988)CrossRefGoogle Scholar
Gescheider, G.: Psychophysics: The Fundamentals. Lawrence Erlbaum Associates, 3rd edn. (1997)Google Scholar
McGee, M.: Usability magnitude estimation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 47(4), 691–695 (2003)CrossRefGoogle Scholar
Moskowitz, H.R.: Magnitude estimation: notes on what, how, when, and why to use it. Journal of Food Quality 1(3), 195–227 (1977)CrossRefGoogle Scholar
Sormunen, E.: Liberal relevance criteria of TREC: Counting on negligible documents? In: 25th SIGIR, pp. 324–330. ACM, New York (2002)Google Scholar
Spink, A., Greisdorf, H.: Regions and levels: Measuring and mapping users’ relevance judgments. JASIST 52(2), 161–173 (2001)CrossRefGoogle Scholar
Stevens, S.S.: A metric for the social consensus. Science 151(3710), 530–541 (1966)CrossRefGoogle Scholar
© Springer International Publishing Switzerland 2015